Developing a USGS Legacy Data Inventory to Preserve and Release Historical USGS Data
Legacy data (n) - Information stored in an old or obsolete format or computer system that is, therefore, difficult to access or process. (Business Dictionary, 2016)
For over 135 years, the U.S. Geological Survey has collected diverse information about the natural world and how it interacts with society. Much of this legacy information is one-of-a-kind and in danger of being lost forever through decay of materials, obsolete technology, or staff changes. Several laws and orders require federal agencies to preserve and provide the public access to federally collected scientific information. The information is to be archived in a manner that allows others to examine the materials for new information or interpretations. Data-at-Risk is a systematic way for the USGS to continue efforts to meet the challenge of preserving and making accessible enormous amount of information locked away in inaccessible formats. Data-at-Risk efforts inventory and prioritize inaccessible information and assist with the preservation and release of the information into the public domain. Much of the information the USGS collects has permanent or long-term value to the Nation and the world through its contributions to furthering scientific discovery, public policies, or decisions. These information collections represent observations and events that will never be repeated and warrant preservation for future generations to learn and benefit from them.
Goal: Expand the USGS contribution to scientific discovery and knowledge by demonstrating a long-term approach to inventorying, prioritizing and releasing to the public the wealth of USGS legacy scientific data.
Implement a systematic workflow to create a USGS Legacy Data Inventory that catalogs and describes known USGS legacy data sets.
Develop a methodology to evaluate and prioritize USGS legacy data sets based on USGS mission and program objectives and potential of successful release within USGS records management and open data policies.
Preserve and release select, priority legacy data sets through the USGS IPDS data release workflow
Analyze the time and resources required to preserve/release legacy data and develop estimates to inform future legacy data inventory efforts.
As one of the largest and oldest earth science organizations in the world, the scientific legacy of the USGS is its data, to include, but not limited to images, video, audio files, physical samples, etc., and the scientific knowledge derived from them, gathered over 130 years of research. However, it is widely understood that high-quality data collected and analyzed as part of now completed projects are hidden away in case files, file cabinets and hard drives housed in USGS facilities. Therefore, despite their potential significance to current USGS mission and program research objectives, these “legacy data” are unavailable. In addition, legacy data are by definition at risk of permanent loss or damage because they pre-date current, open-data policies, standards and formats. Risks to legacy data can be technical, such as obsolescence of the data’s storage media and format, or they can be organizational, such as a lack of funding or facility storage. Conveniently, addressing legacy data risks such as these generally results in the science data becoming useable by modern data tools, as well as accessible to the broader scientific community.
Building on past USGS legacy data inventory and preservation projects
USGS has long history of proactively researching and developing solutions to data management needs, including legacy data inventory and preservation. For example, in 1994 USGS was instrumental in establishing the FGDC-CSDGM metadata standard for geospatial scientific data that is still part of the foundation of USGS data management. Today, USGS is a lead agency in establishing meaningful and actionable policies that facilitate data release to the greater, public scientific community. In recent years, CDI has invested in several legacy data inventory and preservation projects, including the “Legacy Data Inventory” project (aka, “Data Mine” 2013-present), which examined the time, resources and workflows needed for science centers to inventory legacy data. Another CDI project, the “North American Bat Data Recovery and Integration” project (2014-present), is preserving previously unavailable bat banding data (1932-1972) and white-nose syndrome disease data and making them available via APIs. Both of these CDI projects were forward-thinking legacy data initiatives, several years ahead of Federal open data policies and mandates.
However, one of the most comprehensive, Bureau-level legacy data preservation efforts was the USGS Data Rescue project, which provided funding, tools, and support to USGS scientists to preserve legacy data sets at imminent risk of permanent loss or damage. A small sample of USGS science data rescued over those eight fiscal years included:
Inventoried, catalogued, indexed, and preserved Famine Early Warning one-of-a-kind, hardcopy maps.
Landsat orphan scenes, totaling over 146,000 were retrieved and processed, allowing the land research community to access previously unavailable satellite records.
Through a partnership with the Alaska State Division of Geological and Geophysical Surveys, the Alaska Water Science Center scanned, added metadata to, and included in a database volcano imagery dating from the 1950s to 2004.
20,000 original, historical stream flow measurements from Kentucky dating from the early 1900s to the late 1980s were scanned and entered into NWIS.
Central Mineral and Environmental Resources Science Center geochemical data conversion totaling approximately 250,000 primary documents from paper to electronic format were completed.
California Water Science Center migrated paper well schedules and other groundwater records dating back more than 100 years old. The records define historical climate variability, geologic conditions where natural hazards occur, and the extents of freshwater resources.
Over 100 projects were supported in the 8 years the Data Rescue project was in operation (2006-2013), while an additional 300 projects went unfunded, providing a glimpse of the potential trove of USGS legacy data at risk of damage or loss. The urgency of and strategies for preserving USGS legacy data have been discussed at length at the 2014 CSAS&L Data Management Workshop and the 2015 CDI Workshop, further emphasizing a Bureau-wide recognition of the importance of legacy data preservation and release. During the 2015 CDI Workshop, legacy data preservation was rated a top-rated FY16 priority by the Data Management Working Group, laying the groundwork for this proposal, which intends to apply the legacy data inventory and evaluation methods developed through the CDI Legacy Data Inventory project to formalize and extend the inventory successfully started through the Data Rescue Program. By creating a formal method to submit, document and evaluate legacy data known to be in need of preservation, USGS would have a tool that USGS scientists, science centers, and mission areas can use to identify significant historical legacy data that can inform, new, data-intensive scientific efforts.
Challenges and improvements for USGS legacy data preservation and release
Based on our experiences managing and preserving USGS legacy data, we have seen two challenges that often undermine legacy data preservation and release:
The most scientifically significant legacy data aren’t always the most recoverable: Legacy data by definition are “dated” because there is some length of time that has passed since the data were collected, the project completed and recovery efforts begin. The longer the time that’s passed, the more likely project staff aren’t available and supporting project and data documents are lost. Lacking this knowledge and/or documentation, metadata may not be completed, resulting in preserved data that aren’t useable - a critical element of the USGS data release peer review and approval process. If data is not useable, it is more difficult to release. Critically evaluating legacy data for their “release potential,” not just their scientific significance, increases the likelihood of selecting legacy data that will be successfully released.
Research scientists may not have data science skills/expertise/resources: Traditionally, legacy data efforts provide funding directly to the data owner, who is generally a principal investigator and knows the data intimately, but may lack the data science experience, time and tools to preserve and release data in an open format with complete, compliant metadata. In our experience, this can lead to delays in preserving and releasing legacy data. Data scientists can/should not replace data owners, but they can provide a significant level of assistance to data owners, by applying their data and metadata development experience and tools.
We believe that each of challenges have good solutions that can improve the efficiency and predictability of preservation and release efforts:
Make “potential for successful release” a primary evaluation factor in prioritizing and selecting legacy data for preservation and release. By developing a method of estimating the feasibility and cost of preserving and releasing data and incorporating it into the evaluation and priority criteria, we can better select and prioritize data sets.
Provide funding to a USGS data scientist to collaborate with data owners and ensure preservation and releases are consistently produced and of the highest quality.
Each objective of this proposal will be addressed in a sequence of 3 phases:
Legacy Data Inventory Submission Period
Evaluation and prioritization of the Legacy Data Inventory; selection of data sets for preservation and release.
Preservation and release of selected datasets.
Phase I: Identification and inventory of USGS data at risk
Data owners will document their legacy data sets electronically, providing the primary project and data set metadata elements needed to score, evaluate and prioritize the legacy data inventory. The core of these metadata elements will be derived from the established “USGS Metadata 20 Questions” form, which has proven effective at gathering metadata from research scientists with little/no data science experience. Narrative fields will be used for evaluating need. Categorical fields will be used to calculate feasibility scores used to determine level of effort required to successfully rescue the proposed data.
Phase II: Evaluation and prioritization of the USGS data at risk requests
The CDI Data Management Working Group’s Data at Risk sub-group will facilitate the evaluation and prioritization of the legacy data inventory. Mission Areas will be engaged to verify inventory submissions are supported programmatically and meet mission objectives. The USGS Records Management Program, Enterprise Publishing Program, and Sciencebase will be consulted to verify submitted legacy data inventory submissions can be released within Bureau records management and data release policies. Once these checkpoints have been verified the Data at Risk sub-group and data scientist will score and prioritize the legacy data inventory based on the following criteria:
Scientific value/significance to USGS mission area and program objectives.
Potential of successfully preserving and releasing the data by the data scientist.
Severity/Imminence of loss or damage to data based on identified risk factors.
Phase III: Preservation and Release of Select, Priority Legacy Data
Working in order of priority as set in Phase II, the data scientist(s) will collaborate with the data owner and work with them to complete the process of preserving and releasing their legacy data. Through this data owner/scientist collaboration, the data scientist will create and validate the FGDC-CSDGM metadata and develop the data set in an open-format as documented in the metadata. By process, the data scientist will act as an agent of the data owner, coordinating and completing all steps in each workflow until the the IPDS record approved and disseminated by the Bureau and the Sciencebase data release item(s) are approved, locked and made public by the Sciencebase team. However, while the data scientist is responsible for ensuring all preservation and release tasks are completed consistently and within policies and best practices, the data owner retains all approval of final metadata attribution (e.g., title, authorship), as well as disposition of their legacy data (e.g., pre/post processing methods; derivative data architectures).
At the completion of Phase III, each legacy data release will have the following created by the data scientist:
complete, compliant FGDC-CSDGM metadata
legacy data set(s) in an open-format, publicly discoverable and available from Sciencebase.
a USGS highlight submitted through the SW Region to Reston.
a CDI update describing the data set(s) released and a summary of time and resources required to complete the release.
Investigating Current and Historic White-tailed Kite Population Trends Using Genetic Techniques
A White-tailed kite. Photo by Heather Mohan.
The White‑tailed Kite has demonstrated large population fluctuations over the last 150 years. Once common in California, Texas, and the southeast United States, kite numbers declined to very low levels in the 1900s and was thought to be on the verge of extinction by the 1930s. In the 1940s populations began to expand and increasing numbers were observed during the following decades. Today, the White-tailed Kites are common residents throughout much of California, with slow but steady increases in population numbers in the Central Plains states, Texas and Florida. It is unknown whether current U.S. populations suffered a severe genetic bottleneck in the early 1900s and have rebounded since, or whether current U.S. population growth has been related to immigration from Central and South America populations. We are using genetic techniques to examine museum specimens collected before 1930 and modern samples collected in the 1990s from California to provide clues as to whether modern Kites in California in fact have low diversity due to a genetic bottleneck or whether they were founded and sustained by immigrants from other continents.
Examining Current Subspecies Delineations in White-tailed Ptarmigan Using Genomic Data
A White-tailed Ptarmigan on Mt. Evans, Colorado. Photo by Cameron Aldridge, USGS.
The delineation of populations that are evolutionarily and demographically distinct is an important step in the development of species-specific management plans. Such knowledge is necessary for learning how conservation threats vary across a species’ range, for devising strategies to increase population growth rates, and for providing legal protection at the intraspecific level. It is also essential for conserving long-term evolutionary resilience, given that the genetic diversity that has evolved in response to spatial variation in environmental conditions could provide the raw ingredients necessary to fuel future adaptive evolution. We are using genomic data to delineate distinct evolutionary units across the range of the white-tailed ptarmigan. This information will inform management strategies for this alpine species, which may be vulnerable to climate change.
Dam removal and anadromous salmonid (Oncorhynchus spp.) conservation in California
Quiñones, R.M., T.E. Grantham, B.N. Harvey, J.D. Kiernan, M. Klasson, A.P. Wintzer, and P.B. Moyle
Dam removal is often proposed for restoration of anadromous salmonid populations, which are in serious decline in California. However, the benefits of dam removal vary due to differences in affected populations and potential for environmental impacts. Here, we develop an assessment method to examine the relationship between dam removal and salmonid conservation, focusing on dams that act as complete migration barriers. Specifically, we (1) review the effects of dams on anadromous salmonids, (2) describe factors specific to dam removal in California, (3) propose a method to evaluate dam removal effects on salmonids, (4) apply this method to evaluate 24 dams, and (5) discuss potential effects of removing four dams on the Klamath River. Our flexible rating system can rapidly assess the likely effects of dam removal, as a first step in the prioritization of multiple dam removals. We rated eight dams proposed for removal and compared them with another 16 dams, which are not candidates for removal. Twelve of the 24 dams evaluated had scores that indicated at least a moderate benefit to salmonids following removal. In particular, scores indicated that removal of the four dams on the Klamath River is warranted for salmonid conservation. Ultimately, all dams will be abandoned, removed, or rebuilt even if the timespan is hundreds of years. Thus, periodic evaluation of the environmental benefits of dam removal is needed using criteria such as those presented in this paper.
Water management in mediterranean rivers: a multiregional assessment of physical impacts, ecological responses, and management strategies
For 100 years, California's State Water Resources Control Board and its predecessors have been responsible for allocating available water supplies to beneficial uses, but inaccurate and incomplete accounting of water rights has made the state ill-equipped to satisfy growing societal demands for water supply reliability and healthy ecosystems. Here, we present the first comprehensive evaluation of appropriative water rights to identify where, and to what extent, water has been dedicated to human uses relative to natural supplies. The results show that water right allocations total 400 billion cubic meters, approximately five times the state's mean annual runoff. In the state's major river basins, water rights account for up to 1000% of natural surface water supplies, with the greatest degree of appropriation observed in tributaries to the Sacramento and San Joaquin Rivers and in coastal streams in southern California. Comparisons with water supplies and estimates of actual use indicate substantial uncertainty in how water rights are exercised. In arid regions such as California, over-allocation of surface water coupled with trends of decreasing supply suggest that new water demands will be met by re-allocation from existing uses. Without improvements to the water rights system, growing human and environmental demands portend an intensification of regional water scarcity and social conflict. California's legal framework for managing its water resources is largely compatible with needed reforms, but additional public investment is required to enhance the capacity of the state's water management institutions to effectively track and regulate water rights.
Systematic screening of dams for environmental flow assessment and implementation
Environmental flow protections are crucial to the conservation of freshwater biodiversity in dam-regulated river systems. Nevertheless, the implementation of environmental flows has lagged far behind the pace of river ecosystem alteration. The vast number of dams now in operation and the substantial resources required to modify their operations suggest that a systematic approach is needed to identify rivers potentially benefiting from environmental flows. Here, we develop and apply a screening framework that is focused on the selection of dams that warrant environmental flows on the basis of evidence of hydrologic alteration and ecological impairment. Our approach is illustrated for California, where 181 (of an initial 1400+) dams are identified as high-priority candidates to improve environmental flows for native fish conservation. Given the rapid declines of freshwater biodiversity and pervasive alteration to rivers, this systematic screening approach should be useful for guiding the strategic implementation of environmental flows.
Chemical contaminants, health indicators, and reproductive biomarker responses in fish from the Colorado River and its tributaries
Hinck, J.E., V.S. Blazer, N.D. Denslow, K.R. Echols, T.S. Gross, T.W. May, P.J. Anderson, J.J. Coyle, and D.E. Tillitt