Developing a USGS Legacy Data Inventory to Preserve and Release Historical USGS Data
Legacy data (n) - Information stored in an old or obsolete format or computer system that is, therefore, difficult to access or process. (Business Dictionary, 2016)
For over 135 years, the U.S. Geological Survey has collected diverse information about the natural world and how it interacts with society. Much of this legacy information is one-of-a-kind and in danger of being lost forever through decay of materials, obsolete technology, or staff changes. Several laws and orders require federal agencies to preserve and provide the public access to federally collected scientific information. The information is to be archived in a manner that allows others to examine the materials for new information or interpretations. Data-at-Risk is a systematic way for the USGS to continue efforts to meet the challenge of preserving and making accessible enormous amount of information locked away in inaccessible formats. Data-at-Risk efforts inventory and prioritize inaccessible information and assist with the preservation and release of the information into the public domain. Much of the information the USGS collects has permanent or long-term value to the Nation and the world through its contributions to furthering scientific discovery, public policies, or decisions. These information collections represent observations and events that will never be repeated and warrant preservation for future generations to learn and benefit from them.
Goal: Expand the USGS contribution to scientific discovery and knowledge by demonstrating a long-term approach to inventorying, prioritizing and releasing to the public the wealth of USGS legacy scientific data.
Implement a systematic workflow to create a USGS Legacy Data Inventory that catalogs and describes known USGS legacy data sets.
Develop a methodology to evaluate and prioritize USGS legacy data sets based on USGS mission and program objectives and potential of successful release within USGS records management and open data policies.
Preserve and release select, priority legacy data sets through the USGS IPDS data release workflow
Analyze the time and resources required to preserve/release legacy data and develop estimates to inform future legacy data inventory efforts.
As one of the largest and oldest earth science organizations in the world, the scientific legacy of the USGS is its data, to include, but not limited to images, video, audio files, physical samples, etc., and the scientific knowledge derived from them, gathered over 130 years of research. However, it is widely understood that high-quality data collected and analyzed as part of now completed projects are hidden away in case files, file cabinets and hard drives housed in USGS facilities. Therefore, despite their potential significance to current USGS mission and program research objectives, these “legacy data” are unavailable. In addition, legacy data are by definition at risk of permanent loss or damage because they pre-date current, open-data policies, standards and formats. Risks to legacy data can be technical, such as obsolescence of the data’s storage media and format, or they can be organizational, such as a lack of funding or facility storage. Conveniently, addressing legacy data risks such as these generally results in the science data becoming useable by modern data tools, as well as accessible to the broader scientific community.
Building on past USGS legacy data inventory and preservation projects
USGS has long history of proactively researching and developing solutions to data management needs, including legacy data inventory and preservation. For example, in 1994 USGS was instrumental in establishing the FGDC-CSDGM metadata standard for geospatial scientific data that is still part of the foundation of USGS data management. Today, USGS is a lead agency in establishing meaningful and actionable policies that facilitate data release to the greater, public scientific community. In recent years, CDI has invested in several legacy data inventory and preservation projects, including the “Legacy Data Inventory” project (aka, “Data Mine” 2013-present), which examined the time, resources and workflows needed for science centers to inventory legacy data. Another CDI project, the “North American Bat Data Recovery and Integration” project (2014-present), is preserving previously unavailable bat banding data (1932-1972) and white-nose syndrome disease data and making them available via APIs. Both of these CDI projects were forward-thinking legacy data initiatives, several years ahead of Federal open data policies and mandates.
However, one of the most comprehensive, Bureau-level legacy data preservation efforts was the USGS Data Rescue project, which provided funding, tools, and support to USGS scientists to preserve legacy data sets at imminent risk of permanent loss or damage. A small sample of USGS science data rescued over those eight fiscal years included:
Inventoried, catalogued, indexed, and preserved Famine Early Warning one-of-a-kind, hardcopy maps.
Landsat orphan scenes, totaling over 146,000 were retrieved and processed, allowing the land research community to access previously unavailable satellite records.
Through a partnership with the Alaska State Division of Geological and Geophysical Surveys, the Alaska Water Science Center scanned, added metadata to, and included in a database volcano imagery dating from the 1950s to 2004.
20,000 original, historical stream flow measurements from Kentucky dating from the early 1900s to the late 1980s were scanned and entered into NWIS.
Central Mineral and Environmental Resources Science Center geochemical data conversion totaling approximately 250,000 primary documents from paper to electronic format were completed.
California Water Science Center migrated paper well schedules and other groundwater records dating back more than 100 years old. The records define historical climate variability, geologic conditions where natural hazards occur, and the extents of freshwater resources.
Over 100 projects were supported in the 8 years the Data Rescue project was in operation (2006-2013), while an additional 300 projects went unfunded, providing a glimpse of the potential trove of USGS legacy data at risk of damage or loss. The urgency of and strategies for preserving USGS legacy data have been discussed at length at the 2014 CSAS&L Data Management Workshop and the 2015 CDI Workshop, further emphasizing a Bureau-wide recognition of the importance of legacy data preservation and release. During the 2015 CDI Workshop, legacy data preservation was rated a top-rated FY16 priority by the Data Management Working Group, laying the groundwork for this proposal, which intends to apply the legacy data inventory and evaluation methods developed through the CDI Legacy Data Inventory project to formalize and extend the inventory successfully started through the Data Rescue Program. By creating a formal method to submit, document and evaluate legacy data known to be in need of preservation, USGS would have a tool that USGS scientists, science centers, and mission areas can use to identify significant historical legacy data that can inform, new, data-intensive scientific efforts.
Challenges and improvements for USGS legacy data preservation and release
Based on our experiences managing and preserving USGS legacy data, we have seen two challenges that often undermine legacy data preservation and release:
The most scientifically significant legacy data aren’t always the most recoverable: Legacy data by definition are “dated” because there is some length of time that has passed since the data were collected, the project completed and recovery efforts begin. The longer the time that’s passed, the more likely project staff aren’t available and supporting project and data documents are lost. Lacking this knowledge and/or documentation, metadata may not be completed, resulting in preserved data that aren’t useable - a critical element of the USGS data release peer review and approval process. If data is not useable, it is more difficult to release. Critically evaluating legacy data for their “release potential,” not just their scientific significance, increases the likelihood of selecting legacy data that will be successfully released.
Research scientists may not have data science skills/expertise/resources: Traditionally, legacy data efforts provide funding directly to the data owner, who is generally a principal investigator and knows the data intimately, but may lack the data science experience, time and tools to preserve and release data in an open format with complete, compliant metadata. In our experience, this can lead to delays in preserving and releasing legacy data. Data scientists can/should not replace data owners, but they can provide a significant level of assistance to data owners, by applying their data and metadata development experience and tools.
We believe that each of challenges have good solutions that can improve the efficiency and predictability of preservation and release efforts:
Make “potential for successful release” a primary evaluation factor in prioritizing and selecting legacy data for preservation and release. By developing a method of estimating the feasibility and cost of preserving and releasing data and incorporating it into the evaluation and priority criteria, we can better select and prioritize data sets.
Provide funding to a USGS data scientist to collaborate with data owners and ensure preservation and releases are consistently produced and of the highest quality.
Each objective of this proposal will be addressed in a sequence of 3 phases:
Legacy Data Inventory Submission Period
Evaluation and prioritization of the Legacy Data Inventory; selection of data sets for preservation and release.
Preservation and release of selected datasets.
Phase I: Identification and inventory of USGS data at risk
Data owners will document their legacy data sets electronically, providing the primary project and data set metadata elements needed to score, evaluate and prioritize the legacy data inventory. The core of these metadata elements will be derived from the established “USGS Metadata 20 Questions” form, which has proven effective at gathering metadata from research scientists with little/no data science experience. Narrative fields will be used for evaluating need. Categorical fields will be used to calculate feasibility scores used to determine level of effort required to successfully rescue the proposed data.
Phase II: Evaluation and prioritization of the USGS data at risk requests
The CDI Data Management Working Group’s Data at Risk sub-group will facilitate the evaluation and prioritization of the legacy data inventory. Mission Areas will be engaged to verify inventory submissions are supported programmatically and meet mission objectives. The USGS Records Management Program, Enterprise Publishing Program, and Sciencebase will be consulted to verify submitted legacy data inventory submissions can be released within Bureau records management and data release policies. Once these checkpoints have been verified the Data at Risk sub-group and data scientist will score and prioritize the legacy data inventory based on the following criteria:
Scientific value/significance to USGS mission area and program objectives.
Potential of successfully preserving and releasing the data by the data scientist.
Severity/Imminence of loss or damage to data based on identified risk factors.
Phase III: Preservation and Release of Select, Priority Legacy Data
Working in order of priority as set in Phase II, the data scientist(s) will collaborate with the data owner and work with them to complete the process of preserving and releasing their legacy data. Through this data owner/scientist collaboration, the data scientist will create and validate the FGDC-CSDGM metadata and develop the data set in an open-format as documented in the metadata. By process, the data scientist will act as an agent of the data owner, coordinating and completing all steps in each workflow until the the IPDS record approved and disseminated by the Bureau and the Sciencebase data release item(s) are approved, locked and made public by the Sciencebase team. However, while the data scientist is responsible for ensuring all preservation and release tasks are completed consistently and within policies and best practices, the data owner retains all approval of final metadata attribution (e.g., title, authorship), as well as disposition of their legacy data (e.g., pre/post processing methods; derivative data architectures).
At the completion of Phase III, each legacy data release will have the following created by the data scientist:
complete, compliant FGDC-CSDGM metadata
legacy data set(s) in an open-format, publicly discoverable and available from Sciencebase.
a USGS highlight submitted through the SW Region to Reston.
a CDI update describing the data set(s) released and a summary of time and resources required to complete the release.
Decision Support for Climate Adaptation in the Upper Colorado River Basin: Why Drought Decision Makers Choose to Use Tools (or Not)
Chris M. Morris, Creative Commons.
Adapting to climate change and variability, and their associated impacts, requires integrating scientific information into complex decision making processes. Recognizing this challenge, there have been calls for federal climate change science to be designed and conducted in a way that ensures the research translates into effective decision support. Despite the existence of many decision support tools, however, the factors that influence which decision makers choose to use which decision support tools remain poorly understood. Using the Upper Colorado River Drought Early Warning System as a case study, this research will 1) examine how managers choose between many available tools and 2) consider how tool creators can better align their offerings to decision maker needs.
1. Improve understanding of:
The factors that influence decision makers’ choices to use decision support tools or not, and how they choose between available tools
How scientists creating decision support tools currently interface with decision makers and how their outreach efforts do or do not match information channels preferred by managers
The role that decision support tools play in drought decision making
2. Provide useful information to the National Integrated Drought Information System about the current use of the Upper Colorado River Basin Drought Early Warning System
Study Area and Scope
The Upper Colorado River Basin (UCRB) was one of the first pilot areas, beginning in 2008, for implementation of a regional drought early warning system (DEWS) under the National Integrated Drought Information System (NIDIS), which now supports ten regional DEWS. The selection of the UCRB for a DEWS reflects the regional importance of drought monitoring for managing water supply for agriculture and other uses, and the need for effective decision support related to drought. New drought-information tools have been developed specifically for the UCRB DEWS, and a number of others have been created since 2008, adding to the pre-existing toolkit for drought decision making. The various tools that are now available in the UCRB region can be expected to be more or less suitable for different decision makers’ needs. As a result, the broad decision context of this case study (managing drought) is fixed, but information needs vary. Thus decision makers will make varied choices about which of the available tools to use or not use.
The overall aim is to juxtapose understanding of the tool development process of tool creators with understanding of the choices made by prospective tool users to incorporate (or not) given decision support tools into their drought decision making. Document analysis will provide context and an official view of tool development or agency decision making. Conversations with scientists creating tools and drought decision makers will be used to understand motivations, priorities, concerns, and tacit influences on behavior.
Examining Current Subspecies Delineations in White-tailed Ptarmigan Using Genomic Data
A White-tailed Ptarmigan on Mt. Evans, Colorado. Photo by Cameron Aldridge, USGS.
The delineation of populations that are evolutionarily and demographically distinct is an important step in the development of species-specific management plans. Such knowledge is necessary for learning how conservation threats vary across a species’ range, for devising strategies to increase population growth rates, and for providing legal protection at the intraspecific level. It is also essential for conserving long-term evolutionary resilience, given that the genetic diversity that has evolved in response to spatial variation in environmental conditions could provide the raw ingredients necessary to fuel future adaptive evolution. We are using genomic data to delineate distinct evolutionary units across the range of the white-tailed ptarmigan. This information will inform management strategies for this alpine species, which may be vulnerable to climate change.
Patterns and causes of observed piñon pine mortality in the southwestern United States
Meddens, A.J.H., J.A. Hicke, A.K. Macalady, P.C. Buotte, T.R. Cowles, and C.D. Allen
Recently, widespread piñon pine die-off occurred in the southwestern United States. Here we synthesize observational studies of this event and compare findings to expected relationships with biotic and abiotic factors. Agreement exists on the occurrence of drought, presence of bark beetles and increased mortality of larger trees. However, studies disagree about the influences of stem density, elevation and other factors, perhaps related to study design, location and impact of extreme drought. Detailed information about bark beetles is seldom reported and their role is poorly understood. Our analysis reveals substantial limits to our knowledge regarding the processes that produce mortality patterns across space and time, indicating a poor ability to forecast mortality in response to expected increases in future droughts.
Chemical contaminants, health indicators, and reproductive biomarker responses in fish from the Colorado River and its tributaries
Hinck, J.E., V.S. Blazer, N.D. Denslow, K.R. Echols, T.S. Gross, T.W. May, P.J. Anderson, J.J. Coyle, and D.E. Tillitt
The Greater Sage-Grouse, has been observed, hunted, and counted for decades. The sagebrush biome, home to the Greater Sage-Grouse, includes sagebrush-steppe and Great Basin sagebrush communities, interspersed with grasslands, salt flats, badlands, mountain ranges, springs, intermittent creeks and washes, and major river systems, and is one of the most widespread and enigmatic components of Western U.S. landscapes. Over time, habitat conversion, degradation, and fragmentation have accumulated across the entire range such that local conditions as well as habitat distributions at local and regional scales are negatively affecting the long-term persistence of this species. Historic patterns of human use and settlement of the sagebrush ecosystem have contributed to the current condition and status of sage-grouse populations. The accumulation of habitat loss, persistent habitat degradation, and fragmentation by industry and urban infrastructure, as indicated by U.S. Fish and Wildlife Service (USFWS) findings, presents a significant challenge for conservation of this species and sustainable management of the sagebrush ecosystem. Because of the wide variations in natural and human history across these landscapes, no single prescription for management of sagebrush ecosystems (including sage-grouse habitats) will suffice to guide the collective efforts of public and private entities to conserve the species and its habitat.
This report documents and summarizes several decades of work on sage-grouse populations, sagebrush as habitat, and sagebrush community and ecosystem functions based on the recent assessment and findings of the USFWS under consideration of the Endangered Species Act. As reflected here, some of these topics receive a greater depth of discussion because of the perceived importance of the issue for sagebrush ecosystems and sage-grouse populations. Drawing connections between the direct effects on sagebrush ecosystems and the effect of ecosystem condition on habitat condition, and finally the connection between habitat quality and sage-grouse population dynamics remains an important goal for science, management, and conservation. This effort is necessary, despite the perception that these complicated, indirect relations are difficult to characterize and manage, and the many advances in understanding and application developed toward this end have been documented here to help inform regional planning and policy decisions.
Temperature as a potent driver of regional forest drought stress and tree mortality
Williams, A.P., C.D. Allen, A.K. Macalady, D. Griffin, C.A. Woodhouse, D.M. Meko, T.W. Swetnam, S.A. Rauscher, R. Seager, H.D. Grissino-Mayer, J.S. Dean, E.R. Cook, C. Gangodagamage, M. Cai, and N.G. McDowell