USGS Home
Contact USGS
Search USGS

Accuracy assessment/validation methodology and results of 2010–11 land-cover/land-use data for Pools 13, 26, La Grange, and Open River South, Upper Mississippi River System

Jakusz, J.W., Dieck, J.J., Langrehr, H.A., Ruhser, J.J., and Lubinski, S.J., 2015, Accuracy assessment/validation methodology and results of 2010–11 land-cover/land-use data for Pools 13, 26, La Grange, and Open River South, Upper Mississippi River System: U.S. Army Corps of Engineers’ Upper Mississippi River Restoration Program, Long Term Resource Monitoring Technical Report 2015–T001 39 p., including appendixes 1–18, http://pubs.er.usgs.gov/publication/70159276.

Abstract

In 2008, the Upper Mississippi River Restoration (UMRR) Program reached a collaborative agreement with the U.S. Fish and Wildlife Service-Region 3 to collect high-resolution digital imagery of the entire UMRS floodplain during 2010–11 for Long Term Resource Monitoring (LTRM) element. The USGS-Upper Midwest Environmental Sciences Center helped acquire, process, and serve this imagery, as well as produce and serve the 2010–11 Land Cover Use (LCU) systemic dataset of the Upper Mississippi River System floodplain.

While the 1989 and 2000 LCU systemic datasets have not gone through a traditional thematic accuracy assessment (AA) in the past, nor have they undergone a validation analysis, the end products are of high quality. For each systemic dataset produced (1989, 2000, 2010–11), extensive field reconnaissance is performed before photointerpretation. The intent of this field reconnaissance is to learn, test, and verify image signatures as they relate to vegetation types. Questionable areas on the imagery are visited, and the plants or land features observed in the area are recorded for reference. This procedure verifies vegetation signatures on the imagery with those on the ground. In addition, once the photointerpretation is complete, the final LCU dataset undergoes extensive quality assurance/quality control to ensure the imagery is mapped correctly.

The objective of an AA is to measure the probability that a particular location has been assigned its correct vegetation class. An AA estimates thematic (map class) errors in the data, giving users information needed to determine data suitability for a particular application. At the same time, data producers are able to learn more about the nature of errors in the data. Thus, the two attributes of an AA are “producers’ accuracy,” which is the probability that an AA point has been mapped correctly (also referred to as an error of omission); and “users’ accuracy,” which is the probability that the map actually represents what was found on the ground (also referred to as error of commission). Producers’ and users’ accuracies can be obtained from the same set of data by using different analyses.

Accuracy assessment is an extensive effort that requires seasonal field personnel and equipment, data entry, analyses, and post processing—tasks that are costly and time consuming. The geospatial team at the UMESC has suggested a validation process for understanding the accuracy of the spatial datasets, which will be tested on at least some areas of the UMRS. Validation is not a true verification of map-class type in the field; however, it can provide the user of the map with useful information that is similar to a field AA.

Similar to an AA, validation involves generating random points based on the total area for each map class. However, instead of collecting field data, two or three individuals not involved with the photo-interpretative mapping separately review each of the points onscreen and record a best-fit vegetation type(s) for each site. Once the individual analyses are complete, results are joined together and a comparative analysis is performed. The objective of this initial analysis is to identify areas where the validation results were in agreement (matches) and areas where validation results were in disagreement (mismatches). The two or three individuals then perform an analysis, looking at each mismatched site, and agree upon a final validation class. (If two vegetation types at a specific site appear to be equally prevalent, the validation team is permitted to assign the site two best-fit vegetation types.) Following the validation team’s comparative analysis of vegetation assignments, the data are entered into a database and compared to the mappers’ vegetation assignments. Agreements and disagreements between the map and validation classes are identified, and a contingency table is produced. This document presents the AA processes/results for Pools 13 and La Grange, as well as the validation process/results for Pools 13 and 26 and Open River South.

It is important that decisions to undertake AA or validation in the future are made with an understanding of how the advantages and limitations of each strategy align with the specific goals of the project. Validation departs from true AA in that there is not a comparison with vegetation data collected on the ground by an independent field crew. If future accuracy efforts on the UMRS LCU data are performed, traditional thematic AA would be recommended.

Keywords