Developing a USGS Legacy Data Inventory to Preserve and Release Historical USGS Data
Legacy data (n) - Information stored in an old or obsolete format or computer system that is, therefore, difficult to access or process. (Business Dictionary, 2016)
For over 135 years, the U.S. Geological Survey has collected diverse information about the natural world and how it interacts with society. Much of this legacy information is one-of-a-kind and in danger of being lost forever through decay of materials, obsolete technology, or staff changes. Several laws and orders require federal agencies to preserve and provide the public access to federally collected scientific information. The information is to be archived in a manner that allows others to examine the materials for new information or interpretations. Data-at-Risk is a systematic way for the USGS to continue efforts to meet the challenge of preserving and making accessible enormous amount of information locked away in inaccessible formats. Data-at-Risk efforts inventory and prioritize inaccessible information and assist with the preservation and release of the information into the public domain. Much of the information the USGS collects has permanent or long-term value to the Nation and the world through its contributions to furthering scientific discovery, public policies, or decisions. These information collections represent observations and events that will never be repeated and warrant preservation for future generations to learn and benefit from them.
Goal: Expand the USGS contribution to scientific discovery and knowledge by demonstrating a long-term approach to inventorying, prioritizing and releasing to the public the wealth of USGS legacy scientific data.
Implement a systematic workflow to create a USGS Legacy Data Inventory that catalogs and describes known USGS legacy data sets.
Develop a methodology to evaluate and prioritize USGS legacy data sets based on USGS mission and program objectives and potential of successful release within USGS records management and open data policies.
Preserve and release select, priority legacy data sets through the USGS IPDS data release workflow
Analyze the time and resources required to preserve/release legacy data and develop estimates to inform future legacy data inventory efforts.
As one of the largest and oldest earth science organizations in the world, the scientific legacy of the USGS is its data, to include, but not limited to images, video, audio files, physical samples, etc., and the scientific knowledge derived from them, gathered over 130 years of research. However, it is widely understood that high-quality data collected and analyzed as part of now completed projects are hidden away in case files, file cabinets and hard drives housed in USGS facilities. Therefore, despite their potential significance to current USGS mission and program research objectives, these “legacy data” are unavailable. In addition, legacy data are by definition at risk of permanent loss or damage because they pre-date current, open-data policies, standards and formats. Risks to legacy data can be technical, such as obsolescence of the data’s storage media and format, or they can be organizational, such as a lack of funding or facility storage. Conveniently, addressing legacy data risks such as these generally results in the science data becoming useable by modern data tools, as well as accessible to the broader scientific community.
Building on past USGS legacy data inventory and preservation projects
USGS has long history of proactively researching and developing solutions to data management needs, including legacy data inventory and preservation. For example, in 1994 USGS was instrumental in establishing the FGDC-CSDGM metadata standard for geospatial scientific data that is still part of the foundation of USGS data management. Today, USGS is a lead agency in establishing meaningful and actionable policies that facilitate data release to the greater, public scientific community. In recent years, CDI has invested in several legacy data inventory and preservation projects, including the “Legacy Data Inventory” project (aka, “Data Mine” 2013-present), which examined the time, resources and workflows needed for science centers to inventory legacy data. Another CDI project, the “North American Bat Data Recovery and Integration” project (2014-present), is preserving previously unavailable bat banding data (1932-1972) and white-nose syndrome disease data and making them available via APIs. Both of these CDI projects were forward-thinking legacy data initiatives, several years ahead of Federal open data policies and mandates.
However, one of the most comprehensive, Bureau-level legacy data preservation efforts was the USGS Data Rescue project, which provided funding, tools, and support to USGS scientists to preserve legacy data sets at imminent risk of permanent loss or damage. A small sample of USGS science data rescued over those eight fiscal years included:
Inventoried, catalogued, indexed, and preserved Famine Early Warning one-of-a-kind, hardcopy maps.
Landsat orphan scenes, totaling over 146,000 were retrieved and processed, allowing the land research community to access previously unavailable satellite records.
Through a partnership with the Alaska State Division of Geological and Geophysical Surveys, the Alaska Water Science Center scanned, added metadata to, and included in a database volcano imagery dating from the 1950s to 2004.
20,000 original, historical stream flow measurements from Kentucky dating from the early 1900s to the late 1980s were scanned and entered into NWIS.
Central Mineral and Environmental Resources Science Center geochemical data conversion totaling approximately 250,000 primary documents from paper to electronic format were completed.
California Water Science Center migrated paper well schedules and other groundwater records dating back more than 100 years old. The records define historical climate variability, geologic conditions where natural hazards occur, and the extents of freshwater resources.
Over 100 projects were supported in the 8 years the Data Rescue project was in operation (2006-2013), while an additional 300 projects went unfunded, providing a glimpse of the potential trove of USGS legacy data at risk of damage or loss. The urgency of and strategies for preserving USGS legacy data have been discussed at length at the 2014 CSAS&L Data Management Workshop and the 2015 CDI Workshop, further emphasizing a Bureau-wide recognition of the importance of legacy data preservation and release. During the 2015 CDI Workshop, legacy data preservation was rated a top-rated FY16 priority by the Data Management Working Group, laying the groundwork for this proposal, which intends to apply the legacy data inventory and evaluation methods developed through the CDI Legacy Data Inventory project to formalize and extend the inventory successfully started through the Data Rescue Program. By creating a formal method to submit, document and evaluate legacy data known to be in need of preservation, USGS would have a tool that USGS scientists, science centers, and mission areas can use to identify significant historical legacy data that can inform, new, data-intensive scientific efforts.
Challenges and improvements for USGS legacy data preservation and release
Based on our experiences managing and preserving USGS legacy data, we have seen two challenges that often undermine legacy data preservation and release:
The most scientifically significant legacy data aren’t always the most recoverable: Legacy data by definition are “dated” because there is some length of time that has passed since the data were collected, the project completed and recovery efforts begin. The longer the time that’s passed, the more likely project staff aren’t available and supporting project and data documents are lost. Lacking this knowledge and/or documentation, metadata may not be completed, resulting in preserved data that aren’t useable - a critical element of the USGS data release peer review and approval process. If data is not useable, it is more difficult to release. Critically evaluating legacy data for their “release potential,” not just their scientific significance, increases the likelihood of selecting legacy data that will be successfully released.
Research scientists may not have data science skills/expertise/resources: Traditionally, legacy data efforts provide funding directly to the data owner, who is generally a principal investigator and knows the data intimately, but may lack the data science experience, time and tools to preserve and release data in an open format with complete, compliant metadata. In our experience, this can lead to delays in preserving and releasing legacy data. Data scientists can/should not replace data owners, but they can provide a significant level of assistance to data owners, by applying their data and metadata development experience and tools.
We believe that each of challenges have good solutions that can improve the efficiency and predictability of preservation and release efforts:
Make “potential for successful release” a primary evaluation factor in prioritizing and selecting legacy data for preservation and release. By developing a method of estimating the feasibility and cost of preserving and releasing data and incorporating it into the evaluation and priority criteria, we can better select and prioritize data sets.
Provide funding to a USGS data scientist to collaborate with data owners and ensure preservation and releases are consistently produced and of the highest quality.
Each objective of this proposal will be addressed in a sequence of 3 phases:
Legacy Data Inventory Submission Period
Evaluation and prioritization of the Legacy Data Inventory; selection of data sets for preservation and release.
Preservation and release of selected datasets.
Phase I: Identification and inventory of USGS data at risk
Data owners will document their legacy data sets electronically, providing the primary project and data set metadata elements needed to score, evaluate and prioritize the legacy data inventory. The core of these metadata elements will be derived from the established “USGS Metadata 20 Questions” form, which has proven effective at gathering metadata from research scientists with little/no data science experience. Narrative fields will be used for evaluating need. Categorical fields will be used to calculate feasibility scores used to determine level of effort required to successfully rescue the proposed data.
Phase II: Evaluation and prioritization of the USGS data at risk requests
The CDI Data Management Working Group’s Data at Risk sub-group will facilitate the evaluation and prioritization of the legacy data inventory. Mission Areas will be engaged to verify inventory submissions are supported programmatically and meet mission objectives. The USGS Records Management Program, Enterprise Publishing Program, and Sciencebase will be consulted to verify submitted legacy data inventory submissions can be released within Bureau records management and data release policies. Once these checkpoints have been verified the Data at Risk sub-group and data scientist will score and prioritize the legacy data inventory based on the following criteria:
Scientific value/significance to USGS mission area and program objectives.
Potential of successfully preserving and releasing the data by the data scientist.
Severity/Imminence of loss or damage to data based on identified risk factors.
Phase III: Preservation and Release of Select, Priority Legacy Data
Working in order of priority as set in Phase II, the data scientist(s) will collaborate with the data owner and work with them to complete the process of preserving and releasing their legacy data. Through this data owner/scientist collaboration, the data scientist will create and validate the FGDC-CSDGM metadata and develop the data set in an open-format as documented in the metadata. By process, the data scientist will act as an agent of the data owner, coordinating and completing all steps in each workflow until the the IPDS record approved and disseminated by the Bureau and the Sciencebase data release item(s) are approved, locked and made public by the Sciencebase team. However, while the data scientist is responsible for ensuring all preservation and release tasks are completed consistently and within policies and best practices, the data owner retains all approval of final metadata attribution (e.g., title, authorship), as well as disposition of their legacy data (e.g., pre/post processing methods; derivative data architectures).
At the completion of Phase III, each legacy data release will have the following created by the data scientist:
complete, compliant FGDC-CSDGM metadata
legacy data set(s) in an open-format, publicly discoverable and available from Sciencebase.
a USGS highlight submitted through the SW Region to Reston.
a CDI update describing the data set(s) released and a summary of time and resources required to complete the release.
Effects of hypoxia on consumption, growth, and RNA:DNA ratios of young Yellow Perch
Roberts, J.J., S.B. Brandt, D. Fanslow, S.A. Ludsin, S.A. Pothoven, D. Scavia, T.O. Höök
Hypoxia occurs seasonally in many stratified coastal marine and freshwater ecosystems when bottom dissolved oxygen (DO) concentrations are depleted below 2–3 mg O2 L−1.
We evaluated the effects of hypoxia on fish habitat quality in the central basin of Lake Erie from 1987 to 2005, using bioenergetic growth rate potential (GRP) as a proxy for habitat quality. We compared the effect of hypoxia on habitat quality of (i) rainbow smelt, Osmerus mordax mordax Mitchill (young-of-year, YOY, and adult), a cold-water planktivore, (ii) emerald shiner, Notropis atherinoides Rafinesque (adult), a warm-water planktivore, (iii) yellow perch, Perca flavescens Mitchill (YOY and adult), a cool-water benthopelagic omnivore and (iv) round goby Neogobius melanostomus Pallas (adult) a eurythermal benthivore. Annual thermal and DO profiles were generated from 1D thermal and DO hydrodynamics models developed for Lake Erie’s central basin.
Hypoxia occurred annually, typically from mid-July to mid-October, which spatially and temporally overlaps with otherwise high benthic habitat quality. Hypoxia reduced the habitat quality across fish species and life stages, but the magnitude of the reduction varied both among and within species because of the differences in tolerance to low DO levels and warm-water temperatures.
Across years, trends in habitat quality mirrored trends in phosphorus concentration and water column oxygen demand in central Lake Erie. The per cent reduction in habitat quality owing to hypoxia was greatest for adult rainbow smelt and round goby (mean: −35%), followed by adult emerald shiner (mean: −12%), YOY rainbow smelt (mean: −10%) and YOY and adult yellow perch (mean: −8.5%).
Our results highlight the importance of differential spatiotemporally interactive effects of DO and temperature on relative fish habitat quality and quantity. These effects have the potential to influence the performance of individual fish species as well as population dynamics, trophic interactions and fish community structure.
ASPN is a Web-based decision tool that assists natural resource managers and planners in identifying and prioritizing social and economic planning issues, and provides guidance on appropriate social and economic methods to address their identified issues.
ASPN covers the breadth of issues facing natural resource management agencies so it is widely applicable for various resources, plans, and projects.
ASPN also realistically accounts for budget and planning time constraints by providing estimated costs and time lengths needed for each of the possible social and economic methods.
ASPN is a valuable starting point for natural resource managers and planners to start working with their agencies’ social and economic specialists. Natural resource management actions have social and economic effects that often require appropriate analyses. Additionally, in the United States, Federal agencies are legally mandated to follow guidance under the National Environmental Policy Act (NEPA), which requires addressing social and economic effects for actions that may cause biophysical impacts. Most natural resource managers and planners lack training in understanding the full range of potential social and economic effects of a management decision as well as an understanding of the variety of methods and analyses available to address these effects. Thus, ASPN provides a common framework which provides consistency within and across natural resource management agencies to assist in identification of pertinent social and economic issues while also allowing the social and economic analyses to be tailored to best meet the needs of the specific plan or project.
ASPN can be used throughout a planning process or be used as a tool to identify potential issues that may be applicable to future management actions. ASPN is useful during the pre-scoping phase as a tool to start thinking about potential social and economic issues as well as to identify potential stakeholders who may be affected. Thinking about this early in the planning process can help with outreach efforts and with understanding the cost and time needed to address the potential social and economic effects. One can use ASPN during the scoping and post-scoping phases as a way to obtain guidance on how to address issues that stakeholders identified. ASPN can also be used as a monitoring tool to identify whether new social and economic issues arise after a management action occurs.
ASPN is developed through a collaborative research effort between the USGS Fort Collins Science Center’s (FORT) Social and Economic Analysis (SEA) Branch and the U.S. Forest Service, the National Park Service, the Bureau of Land Management, and the U.S. Fish and Wildlife Service. ASPN’s technical development is led by the USGS FORT’s Information Science Branch. An updated release, which will extend ASPN’s functionality and incorporate feature improvements identified in ongoing usability testing, is currently in the planning stages.
White-nose syndrome (WNS) is an emerging and devastating disease of hibernating bats in North America. WNS is caused by a cold-growing fungus (Geomyces destructans) that infects the skin of hibernating bats during winter and causes life-threatening alterations in physiology and behavior. WNS has spread rapidly across the eastern United States and Canada since it was first documented in New York in the winter of 2006. This new disease is causing mass mortality and detrimentally affecting most of the 6 species of bats that hibernate in the northeastern United States. Particularly hard-hit are the little brown bat (Myotis lucifugus), northern long-eared bat (Myotis septentrionalis), eastern small-footed bat (Myotis leibii), and federally endangered Indiana bat (Myotis sodalis). Several more species are also now known to be exposed to the fungus in the Midwest and Southeast. The sudden and widespread mortality associated with white-nose syndrome is unprecedented in any of the world’s bats and is a cause for international concern as the fungus and the disease spread farther north, south, and west. Loss of these long-lived insect-eating bats could have substantial adverse effects on agriculture and forestry through loss of natural pest-control services.
Tracking a Deadly Disease
Because WNS is spreading so rapidly, field surveillance data and diagnostic samples must be managed efficiently so that critical information can be communicated quickly among State and Federal land managers, as well as the public. The U.S. Fish and Wildlife Service, which plays a primary role in coordinating the Federal response to WNS, worked with the USGS Fort Collins Science Center’s Web Applications Team to develop the White-nose Syndrome Disease Tracking System. Version 1.0 of this system, released for Beta testing in May 2011, addresses two critical objectives:
enable state-level resource managers to effectively manage WNS field and laboratory data, and
provide customizable map and data reports of surveillance findings. The WNS Disease Tracking System subsequently was demonstrated to resource managers involved in the WNS response, and system users are assisting with in-depth testing. Once resource-management users are all trained (autumn 2011), they will begin populating the system with surveillance data, much of which will be immediately available to the public.
WNS version 1.0 was released into production in November, 2011 and state points-of-contact are currently being trainined. New users are providing ciritical feedback for WNS version 2.0, which is currently being planned with Fish and Wildlife Region 5 and the National White-nose Syndrome Data Management Team.
Key System Components
Disease Tracking: Customizable disease tracking maps and data exports for all U.S. states and counties
Disease Reporting: Tissue sample database management for authorized resource managers as well as a publicly accessible database of disease reporting contacts for all U.S. States and Federal resource management agencies
Diagnostic Labs: Directory of laboratories involved in white-nose syndrome diagnostic analyses
Spatial and temporal patterns in hydrochemistry of a coastal marsh in Saginaw Bay, Lake Huron, U.S.A.