Developing a USGS Legacy Data Inventory to Preserve and Release Historical USGS Data
Legacy data (n) - Information stored in an old or obsolete format or computer system that is, therefore, difficult to access or process. (Business Dictionary, 2016)
For over 135 years, the U.S. Geological Survey has collected diverse information about the natural world and how it interacts with society. Much of this legacy information is one-of-a-kind and in danger of being lost forever through decay of materials, obsolete technology, or staff changes. Several laws and orders require federal agencies to preserve and provide the public access to federally collected scientific information. The information is to be archived in a manner that allows others to examine the materials for new information or interpretations. Data-at-Risk is a systematic way for the USGS to continue efforts to meet the challenge of preserving and making accessible enormous amount of information locked away in inaccessible formats. Data-at-Risk efforts inventory and prioritize inaccessible information and assist with the preservation and release of the information into the public domain. Much of the information the USGS collects has permanent or long-term value to the Nation and the world through its contributions to furthering scientific discovery, public policies, or decisions. These information collections represent observations and events that will never be repeated and warrant preservation for future generations to learn and benefit from them.
Goal: Expand the USGS contribution to scientific discovery and knowledge by demonstrating a long-term approach to inventorying, prioritizing and releasing to the public the wealth of USGS legacy scientific data.
Implement a systematic workflow to create a USGS Legacy Data Inventory that catalogs and describes known USGS legacy data sets.
Develop a methodology to evaluate and prioritize USGS legacy data sets based on USGS mission and program objectives and potential of successful release within USGS records management and open data policies.
Preserve and release select, priority legacy data sets through the USGS IPDS data release workflow
Analyze the time and resources required to preserve/release legacy data and develop estimates to inform future legacy data inventory efforts.
As one of the largest and oldest earth science organizations in the world, the scientific legacy of the USGS is its data, to include, but not limited to images, video, audio files, physical samples, etc., and the scientific knowledge derived from them, gathered over 130 years of research. However, it is widely understood that high-quality data collected and analyzed as part of now completed projects are hidden away in case files, file cabinets and hard drives housed in USGS facilities. Therefore, despite their potential significance to current USGS mission and program research objectives, these “legacy data” are unavailable. In addition, legacy data are by definition at risk of permanent loss or damage because they pre-date current, open-data policies, standards and formats. Risks to legacy data can be technical, such as obsolescence of the data’s storage media and format, or they can be organizational, such as a lack of funding or facility storage. Conveniently, addressing legacy data risks such as these generally results in the science data becoming useable by modern data tools, as well as accessible to the broader scientific community.
Building on past USGS legacy data inventory and preservation projects
USGS has long history of proactively researching and developing solutions to data management needs, including legacy data inventory and preservation. For example, in 1994 USGS was instrumental in establishing the FGDC-CSDGM metadata standard for geospatial scientific data that is still part of the foundation of USGS data management. Today, USGS is a lead agency in establishing meaningful and actionable policies that facilitate data release to the greater, public scientific community. In recent years, CDI has invested in several legacy data inventory and preservation projects, including the “Legacy Data Inventory” project (aka, “Data Mine” 2013-present), which examined the time, resources and workflows needed for science centers to inventory legacy data. Another CDI project, the “North American Bat Data Recovery and Integration” project (2014-present), is preserving previously unavailable bat banding data (1932-1972) and white-nose syndrome disease data and making them available via APIs. Both of these CDI projects were forward-thinking legacy data initiatives, several years ahead of Federal open data policies and mandates.
However, one of the most comprehensive, Bureau-level legacy data preservation efforts was the USGS Data Rescue project, which provided funding, tools, and support to USGS scientists to preserve legacy data sets at imminent risk of permanent loss or damage. A small sample of USGS science data rescued over those eight fiscal years included:
Inventoried, catalogued, indexed, and preserved Famine Early Warning one-of-a-kind, hardcopy maps.
Landsat orphan scenes, totaling over 146,000 were retrieved and processed, allowing the land research community to access previously unavailable satellite records.
Through a partnership with the Alaska State Division of Geological and Geophysical Surveys, the Alaska Water Science Center scanned, added metadata to, and included in a database volcano imagery dating from the 1950s to 2004.
20,000 original, historical stream flow measurements from Kentucky dating from the early 1900s to the late 1980s were scanned and entered into NWIS.
Central Mineral and Environmental Resources Science Center geochemical data conversion totaling approximately 250,000 primary documents from paper to electronic format were completed.
California Water Science Center migrated paper well schedules and other groundwater records dating back more than 100 years old. The records define historical climate variability, geologic conditions where natural hazards occur, and the extents of freshwater resources.
Over 100 projects were supported in the 8 years the Data Rescue project was in operation (2006-2013), while an additional 300 projects went unfunded, providing a glimpse of the potential trove of USGS legacy data at risk of damage or loss. The urgency of and strategies for preserving USGS legacy data have been discussed at length at the 2014 CSAS&L Data Management Workshop and the 2015 CDI Workshop, further emphasizing a Bureau-wide recognition of the importance of legacy data preservation and release. During the 2015 CDI Workshop, legacy data preservation was rated a top-rated FY16 priority by the Data Management Working Group, laying the groundwork for this proposal, which intends to apply the legacy data inventory and evaluation methods developed through the CDI Legacy Data Inventory project to formalize and extend the inventory successfully started through the Data Rescue Program. By creating a formal method to submit, document and evaluate legacy data known to be in need of preservation, USGS would have a tool that USGS scientists, science centers, and mission areas can use to identify significant historical legacy data that can inform, new, data-intensive scientific efforts.
Challenges and improvements for USGS legacy data preservation and release
Based on our experiences managing and preserving USGS legacy data, we have seen two challenges that often undermine legacy data preservation and release:
The most scientifically significant legacy data aren’t always the most recoverable: Legacy data by definition are “dated” because there is some length of time that has passed since the data were collected, the project completed and recovery efforts begin. The longer the time that’s passed, the more likely project staff aren’t available and supporting project and data documents are lost. Lacking this knowledge and/or documentation, metadata may not be completed, resulting in preserved data that aren’t useable - a critical element of the USGS data release peer review and approval process. If data is not useable, it is more difficult to release. Critically evaluating legacy data for their “release potential,” not just their scientific significance, increases the likelihood of selecting legacy data that will be successfully released.
Research scientists may not have data science skills/expertise/resources: Traditionally, legacy data efforts provide funding directly to the data owner, who is generally a principal investigator and knows the data intimately, but may lack the data science experience, time and tools to preserve and release data in an open format with complete, compliant metadata. In our experience, this can lead to delays in preserving and releasing legacy data. Data scientists can/should not replace data owners, but they can provide a significant level of assistance to data owners, by applying their data and metadata development experience and tools.
We believe that each of challenges have good solutions that can improve the efficiency and predictability of preservation and release efforts:
Make “potential for successful release” a primary evaluation factor in prioritizing and selecting legacy data for preservation and release. By developing a method of estimating the feasibility and cost of preserving and releasing data and incorporating it into the evaluation and priority criteria, we can better select and prioritize data sets.
Provide funding to a USGS data scientist to collaborate with data owners and ensure preservation and releases are consistently produced and of the highest quality.
Each objective of this proposal will be addressed in a sequence of 3 phases:
Legacy Data Inventory Submission Period
Evaluation and prioritization of the Legacy Data Inventory; selection of data sets for preservation and release.
Preservation and release of selected datasets.
Phase I: Identification and inventory of USGS data at risk
Data owners will document their legacy data sets electronically, providing the primary project and data set metadata elements needed to score, evaluate and prioritize the legacy data inventory. The core of these metadata elements will be derived from the established “USGS Metadata 20 Questions” form, which has proven effective at gathering metadata from research scientists with little/no data science experience. Narrative fields will be used for evaluating need. Categorical fields will be used to calculate feasibility scores used to determine level of effort required to successfully rescue the proposed data.
Phase II: Evaluation and prioritization of the USGS data at risk requests
The CDI Data Management Working Group’s Data at Risk sub-group will facilitate the evaluation and prioritization of the legacy data inventory. Mission Areas will be engaged to verify inventory submissions are supported programmatically and meet mission objectives. The USGS Records Management Program, Enterprise Publishing Program, and Sciencebase will be consulted to verify submitted legacy data inventory submissions can be released within Bureau records management and data release policies. Once these checkpoints have been verified the Data at Risk sub-group and data scientist will score and prioritize the legacy data inventory based on the following criteria:
Scientific value/significance to USGS mission area and program objectives.
Potential of successfully preserving and releasing the data by the data scientist.
Severity/Imminence of loss or damage to data based on identified risk factors.
Phase III: Preservation and Release of Select, Priority Legacy Data
Working in order of priority as set in Phase II, the data scientist(s) will collaborate with the data owner and work with them to complete the process of preserving and releasing their legacy data. Through this data owner/scientist collaboration, the data scientist will create and validate the FGDC-CSDGM metadata and develop the data set in an open-format as documented in the metadata. By process, the data scientist will act as an agent of the data owner, coordinating and completing all steps in each workflow until the the IPDS record approved and disseminated by the Bureau and the Sciencebase data release item(s) are approved, locked and made public by the Sciencebase team. However, while the data scientist is responsible for ensuring all preservation and release tasks are completed consistently and within policies and best practices, the data owner retains all approval of final metadata attribution (e.g., title, authorship), as well as disposition of their legacy data (e.g., pre/post processing methods; derivative data architectures).
At the completion of Phase III, each legacy data release will have the following created by the data scientist:
complete, compliant FGDC-CSDGM metadata
legacy data set(s) in an open-format, publicly discoverable and available from Sciencebase.
a USGS highlight submitted through the SW Region to Reston.
a CDI update describing the data set(s) released and a summary of time and resources required to complete the release.
Prevalence of the amphibian Chytrid Fungus (Batrachochytrium dendrobatidis) at Buenos Aires National Wildlife Refuge, Arizona, USA
Sigafus, B.H., C.R. Schwalbe, B.R. Hossack, and E. Muths
Unsupported inferences of high-severity fire in historical dry forests of the western United States: response to Williams and Baker
Fulé, P.Z., T.W. Swetnam, P.M. Brown, D.A. Falk, D.L. Peterson, C.D. Allen, G.H. Aplet, M.A. Battaglia, D. Binkley, C. Farris, R.E. Keane, E.Q. Margolis, H. Grissino-Mayer, C. Miller, C. Hull Sieg, C. Skinner, S.L. Stephens, and A. Taylor
Reconstructions of dry western US forests in the late 19th century in Arizona, Colorado and Oregon based on General Land Office records were used by Williams & Baker (2012; Global Ecology and Biogeography, 21, 1042–1052; hereafter W&B) to infer past fire regimes with substantial moderate and high-severity burning. The authors concluded that present-day large, high-severity fires are not distinguishable from historical patterns. We present evidence of important errors in their study. First, the use of tree size distributions to reconstruct past fire severity and extent is not supported by empirical age–size relationships nor by studies that directly quantified disturbance history in these forests. Second, the fire severity classification of W&B is qualitatively different from most modern classification schemes, and is based on different types of data, leading to an inappropriate comparison. Third, we note that while W&B asserted ‘surprising’ heterogeneity in their reconstructions of stand density and species composition, their data are not substantially different from many previous studies which reached very different conclusions about subsequent forest and fire behaviour changes. Contrary to the conclusions of W&B, the preponderance of scientific evidence indicates that conservation of dry forest ecosystems in the western United States and their ecological, social and economic value is not consistent with a present-day disturbance regime of large, high-severity fires, especially under changing climate.
Patterns and causes of observed piñon pine mortality in the southwestern United States
Meddens, A.J.H., J.A. Hicke, A.K. Macalady, P.C. Buotte, T.R. Cowles, and C.D. Allen
Recently, widespread piñon pine die-off occurred in the southwestern United States. Here we synthesize observational studies of this event and compare findings to expected relationships with biotic and abiotic factors. Agreement exists on the occurrence of drought, presence of bark beetles and increased mortality of larger trees. However, studies disagree about the influences of stem density, elevation and other factors, perhaps related to study design, location and impact of extreme drought. Detailed information about bark beetles is seldom reported and their role is poorly understood. Our analysis reveals substantial limits to our knowledge regarding the processes that produce mortality patterns across space and time, indicating a poor ability to forecast mortality in response to expected increases in future droughts.
Chemical contaminants, health indicators, and reproductive biomarker responses in fish from the Colorado River and its tributaries
Hinck, J.E., V.S. Blazer, N.D. Denslow, K.R. Echols, T.S. Gross, T.W. May, P.J. Anderson, J.J. Coyle, and D.E. Tillitt
Recent restrictions on uranium mining within the Grand Canyon watershed have drawn attention to scientific data gaps in evaluating the possible effects of ore extraction to human populations as well as wildlife communities in the area. Tissue contaminant concentrations, one of the most basic data requirements to determine exposure, are not available for biota from any historical or active uranium mines in the region. The Canyon Uranium Mine is under development, providing a unique opportunity to characterize concentrations of uranium and other trace elements, as well as radiation levels in biota, found in the vicinity of the mine before ore extraction begins. Our study objectives were to identify contaminants of potential concern and critical contaminant exposure pathways for ecological receptors; conduct biological surveys to understand the local food web and refine the list of target species (ecological receptors) for contaminant analysis; and collect target species for contaminant analysis prior to the initiation of active mining. Contaminants of potential concern were identified as arsenic, cadmium, chromium, copper, lead, mercury, nickel, selenium, thallium, uranium, and zinc for chemical toxicity and uranium and associated radionuclides for radiation. The conceptual exposure model identified ingestion, inhalation, absorption, and dietary transfer (bioaccumulation or bioconcentration) as critical contaminant exposure pathways. The biological survey of plants, invertebrates, amphibians, reptiles, birds, and small mammals is the first to document and provide ecological information on >200 species in and around the mine site; this study also provides critical baseline information about the local food web. Most of the species documented at the mine are common to ponderosa pine Pinus ponderosa and pinyon–juniper Pinus–Juniperus spp. forests in northern Arizona and are not considered to have special conservation status by state or federal agencies; exceptions are the locally endemic Tusayan flameflower Phemeranthus validulus, the long-legged bat Myotis volans, and the Arizona bat Myotis occultus. The most common vertebrate species identified at the mine site included the Mexican spadefoot toad Spea multiplicata, plateau fence lizard Sceloporus tristichus, violet-green swallow Tachycineta thalassina, pygmy nuthatch Sitta pygmaea, purple martin Progne subis, western bluebird Sialia mexicana, deermouse Peromyscus maniculatus, valley pocket gopher Thomomys bottae, cliff chipmunk Tamias dorsalis, black-tailed jackrabbit Lepus californicus, mule deer Odocoileus hemionus, and elk Cervus canadensis. A limited number of the most common species were collected for contaminant analysis to establish baseline contaminant and radiological concentrations prior to ore extraction. These empirical baseline data will help validate contaminant exposure pathways and potential threats from contaminant exposures to ecological receptors. Resource managers will also be able to use these data to determine the extent to which local species are exposed to chemical and radiation contamination once the mine is operational and producing ore. More broadly, these data could inform resource management decisions on mitigating chemical and radiation exposure of biota at high-grade uranium breccia pipes throughout the Grand Canyon watershed.
Testing of Actinobacteria Isolated from Twelve Western Bat Species Against Pseudogymnoascus destructans: Clues to Potential Bat Natural Defenses
Valdez, E.W., D.E. Northup, N.A. Caimi, A. Porras-Alfaro, A.S. Kooser, D. Buecher, J.M. Young, J.C. Kimble, and K.J. Hughes
Parent Publication Title:
2014 White-Nose Syndrome Workshop, September 8 -12, 2014,St. Louis, Missouri
Rivers and their floodplains worldwide have changed dramatically over the last century because of regulation by dams, flow diversions and channel stabilization. Floodplains no longer inundated by river flows following dam-induced flood reduction comprise large areas of bottomland habitat, but the effects of abandonment on plant communities are not well understood. Using a hydraulic flow model, geomorphic mapping and field surveys, we addressed the following questions along the Bill Williams River, Arizona: (i) What percent of the bottomland do abandoned floodplains comprise? and (ii) Are abandoned floodplains quantitatively different from adjacent xeric and riparian surfaces in terms of vegetation composition and surface sediment? We found that nearly 70% of active channel and floodplain area was abandoned following dam installation. Abandoned floodplains along the Bill Williams River tend to be similar to each other yet distinct from neighbouring habitats: they have been altered physically from their historic state, leading to distinct combinations of surface sediments, hydrology and plant communities. Abandoned floodplains may transition to xeric communities over time but are likely to retain some riparian qualities as long as there is access to relatively shallow ground water. With expected increases in water demand and drying climatic conditions in many regions, these surfaces and associated vegetation will continue to be extensive in riparian landscapes worldwide.
Long-term change along the Colorado River in Grand Canyon National Park (1889-2011)
Webb, R.H., J. Belnap, M.L. Scott, J.M. Friedman and T.C. Esque