High Throughput Computing: A Solution for Scientific Analysis

• High Throughput Computing
• Home
• HTC Computing Times

• HTC Research Applications
• Maximum Entropy
• 2D/3D Hydrodynamic Numerical Modeling
• Species Trend Analysis and the Effects of their Surrounding Environment
• Bioclimatic Variables and Continuous Surface Drought Indices
• Landscape-scale Ecological Modeling
• Geostatistical Simulations and Statistical Analysis

• For More Information Contact
• Michael O’Donnell
• Cameron Aldridge
• Bradley Fedy

Landscape-Scale Ecological Modeling

Background

Researchers at the USGS Fort Collins Science Center (FORT) evaluate landscape-scale characteristics for many natural resource questions. Landscape-scale characteristics can represent naïve statistics (which do not consider chance) of features (e.g., terrain ruggedness, percent cover of a vegetation community type, snow deposition) within a specified kernel size (Figure 1). In other words, kernels of varying scales are passed over remotely sensed or raster datasets, and a statistic is calculated for each cell describing all neighboring cells (Figure 1). These statistical measures are often used in linear and nonlinear models to predict landscape use and avoidance by a species. For example, a moving window applied to a percent cover of herbaceous plants may show species selection for moderate cover with a patch size of 100 meters (m), but that same species may avoid moderate cover with a patch size of 2 kilometers.

Figure 1. Example of results after applying a 6400-m kernel to a terrain roughness index. The raw data (left) figure shows greater variability and detail, while the resulting statistic (right) figure shows a more smoothed or generalized surface. The "Kernel" (middle) describes a moving window of a given shape and size that captures a summary of data values within that window as it traverses a spatial dataset. As depicted above, the circle represents the moving kernel and the lines indicate its path.

Problem

Applying moving windows to datasets can be an extremely long process, and this amount of time increases with increased spatial extents, spatial resolutions, and the kernel size. For example, a moving window of 6400 m, with a spatial resolution of 30 m, and a spatial extent of Wyoming, USA, can require up to a week of processing time using commercial off-the-shelf software. Usually about 4 scales of moving windows and hundreds of datasets are needed to evaluate habitat use. So, to evaluate relationships between a species and its habitat selection, a significant amount of processing time is required before any of the species predictive models can be evaluated for habitat selection.

Solution

We originally ran the analysis described above using GIS software, which required about 1100 hours of processing time. FORT then developed a C++ application that reduced the processing time to 100 hours. Our intent was then to use HTCondor¹ and the C++ application to reduce the processing time to several hours. However, the GIS software we used was significantly enhanced in a newly released version, which reduced the processing time to 1 hour. Because we had not fully implemented HTCondor before completing the project requiring these analyses, we did not re-evaluate HTCondor execution times.

However, in the following more complex or data-intensive scenarios (or a combination of the two), HTC would be applicable:

1. Working at a regional scale (multiple states). Using high-resolution data, we could divide the dataset into appropriately sized tiles. Each tile would represent independent jobs and each tile would be processed on an independent machine. To avoid edge effects (statistical measures are biased on edges of raster datasets because a full-sized kernel is not available), each tile is buffered by the radius of the kernel. Once all tiles are processed, they can then be merged back together into a single dataset. The number and size of tiles is determined by testing a single tile and achieving results that complete within a desired amount of time.
2. Processing thousands of statewide datasets for a single state. In this scenario, each statewide product would represent an independent job, and using HTCondor would permit us to significantly reduce processing time. Therefore, future projects requiring similar analyses will greatly benefit from using the approaches we have described in this example. (One project at FORT required over a thousand statewide datasets using the method described here.)