Research Institute *
Environmental Quality Index (EQI) for Evaluation of NRCS Program Effects Nancy French, Richard Wallace, Robert Shuchman Kevin Wickey Presented at Soil and Water Conservation Society Meeting: Managing Agricultural Landscapes for Environmental Quality, Kansas City, MO October 11-13 2006
*
Formerly with ALTARUM www.altarum.org
Outline of Presentation MI-NRCS/MTRI project introduction and motivation – Project overview – Review of agriculture in Michigan
Evaluation strategy for conservation program effectiveness Development of an environmental quality index for MI-NRCS – – – –
Application of metrics and indices for simplifying complex data Use of metrics and indices in evaluation context Structure of the EQI Plans for further development and implementation
Data needs for the index – Solicit your thoughts on possible data for EQI
2
Overall Goals of the MI-NRCS/MTRI Cooperative Agreement Develop and apply a method for evaluating the environmental effects of NRCS conservation programs at the State level – Develop environmental quality metric to assess impacts of NRCS practices on land, air, and water quality – Take a holistic look at how well the programs have achieved the goal of “conservation”
Develop decision-support tools to: – Enable improved communication of conservation-related information within and outside NRCS – Assist MI-NRCS personnel with data handing and visualization to help them better manage conservation programs
3
Ultimate Goal of MI-NRCS/MTRI Project Program Evaluation Tools
EQI
TATS
Program Management Data Tools
Integrated tools Information Dissemination and Visualization Tools
IMS
Provide useful and valid tools and products to improve MI-NRCS operations and program management 4
Agricultural Products in MI
Top Commodities in Michigan (2004)
Michigan Commodities Ranked First in U.S. (% of U.S. total, 2004)
Milk
Cranberry beans, dry (72.2)
Corn
Cherries, tart (70.0)
Soybeans
Black beans, dry (69.0)
Cattle and calves
Navy beans, dry (45.3)
Hogs
Small red beans, dry (43.4)
Animal bedding /garden plants
Blueberries (35.2)
Woody ornamentals
Cucumbers for pickles (29.4)
Wheat
Light, red kidney beans (26.2)
Sugarbeets
Geraniums, seed and cuttings (22.0)
http://www.michigan.gov/mda/0,1607,7-125-2961_6860_7657---,00.html 5
6
7
8
Evaluation Activities within the MI-NRCS/MTRI Cooperative Agreement
Develop a tool for program evaluation and future assessment (EQI) – Aid management and administration of programs and practices
Support CEAP/Tiffin River Special Emphasis Watershed study – See two presentations at this meeting: Brooks et al. and Schaffer et al.
Ultimately – Improve environmental quality of Michigan as affected by agricultural practices – Protect the Great Lakes
9
Program Evaluation Framework
10
Program Implementation & Confounding Influences
Driven by ProTracts data Data normalization (acres of agriculture, expenditures)
Appropriate approaches for cost-effect analyses (common $ metric)
Effect of land use change, climate change, advanced farming practices, etc. on agricultural systems
Understanding confounding influences is key for proper quantification of program outcomes 11
Measuring Environmental Quality Compare to data on program implementation and farming practices
Account/control for confounding influences Use independent data sources, where possible
12
Environmental Quality Index (EQI) Establish an index baseline for measuring change over time and/or monitoring differences in index scores across geographic units of interest (counties, watersheds)
Develop scaling system that accounts for good program outcomes (achieving program goals) – Based on measures of environmental quality over time – Control for confounding influences – Normalize by extent of agriculture and NRCS program application
Obtain data from inside and outside NRCS – based on program goals – – – – – –
ProTracts, PRS, etc. Remote sensing (land use/land cover, habitat diversity, etc.) EPA, USDA, other federal agencies State agencies: MI-DNR, MI-DEQ, MDA Correlations from literature or in-situ measurements NGO’s: Duck Unlimited, MNFI 13
Metrics and Indices An index is used to provide some measurement (hence, metric) of activity, performance, progress, etc. – Examples: Consumer Price Index, Environmental Sustainability Index (Yale), Index of Consumer Sentiment (UM)
An index attempts to capture a complicated concept (such as, what is the state of the nation’s economy, or how are NRCS programs affecting environmental quality) in a single output measure (or metric)
Indices can be contrasted with measurements of specific phenomenon (how many dollars, species, or tons of soil saved), but they usually contain numerous such measurements as their data input – Yale’s Environmental Sustainability Index, e.g., has 76 underlying input variables within five components
The data are then manipulated by some mathematical function, simple or complex, to arrive at the index value 14
More Indices from the Literature For watersheds – Stream habitat assessment (SHA) • Bank vegetation, bottom substrate, habitats…
– Visual stream assessment (VSA) • Channel, odors, zone width, bank vegetation…
– Riparian vegetation index (RVI) • Focuses on type of cover (tall, woody, barren…)
– Mail survey index (MSI)
For land (soil) – Land quality index (LQI) • Developed by Norfleet and colleagues at NRCS’s Soil Quality Institute • Intended to be a measure of sustainability of soil • Includes ecological and economic indicators
Other – Index of biotic integrity (IBI) • Applied mostly to lakes and streams • Developed uniquely for small geographic extents (hence, resource intensive) 15
Reasons for Having an Index Difficult to see the big picture when faced with lots of measures of many discreet phenomenon
Easier to track change over time along a single dimension (even if it is a virtual dimension, as with an index)
Metrics/indices are much better at facilitating relative comparisons (across programs, across time, etc.) than they are at providing absolute measures of performance
Easier to compare different cases when a single metric is used (such as, is the economy better in country X or Y, or is NRCS more effective in meeting environmental goals in Washtenaw or Hillsdale County?) – Must normalize data – Must account for external variables
16
Metrics and Indices in an Evaluation Context Quantitative evaluation requires a counterfactual basis (i.e., what would the world be like if the program, policy, plan, etc., had NEVER come into being) – Compares the world with “it” and the world without it (where it = the thing being evaluated)—the difference is the impact of it – Catch 22: the world without it does not exist (hence, “counterfactual”) • This is where research design comes into play: use of control groups, longitudinal studies, etc.—all attempts to produce some estimate of the counterfactual
Ideally, an index for evaluation would fit into an overall evaluation design (as a primary outcome of interest)
Implementation of NRCS programs is not based on experimental design, rather it is a voluntary program with regulatory restrictions Therefore we are using the index concept as a tool for tracking program progress 17
Program Goals and Measures MI-NRCS Programs, Goals, & Measurables
Land Retirement
Type
Program Conservation Reserve Program (CRP)
Wetlands Reserve Program (WRP)
Conservation Reserve Enhancement Program (CREP)
Management Years • authorized by the Food Security Act of 1985 • implemented by FSA on behalf of USDA’s Commodity Credit Corporation.
• mandated by the Food Security Act of 1985. • reauthorized in the Farm Security and Rural I A f 2002 (F Bill)
• refinement of the Conservation Reserve Program (CRP)
Wildlife Habitat • began in 1998 Incentives Program (WHIP)
Program Type and Assistance Length of Agreement Type Land Retirement using • Financial rental payments & cost • Technical share 10-15 years
Land Retirement using • easements & cost share • 10 or 30 years or permanent Land Retirement using • annual rental payments & cost share 10-15 years
Assistance using cost share 5-15 years
•
Primary Practice Groups Water Quality Buffers
•
Improve the quality of water
•
•
Wetland Protection & Restoration
•
Control soil erosion
•
Upland Bird Buffers
•
Enhance wildlife habitat
Financial
•
Shallow Water Restoration
•
Technical
• Vegetative Restoration; wetlands enhancements; uplands restoration
•
Financial
•
Working Land
Conservation Security Program (CSP)
Forestry Land Enhancement Program (FLEP)
Forest Incentives Program (FIP)
• Authorized by the Farm Security and Rural Investment Act of 2002
• implemented by DNR Forest Service. • implemented in 2003 under NRCS as part of Title VIII of the 2002 Farm Bill. • replaces the Stewardship Incentives Program (SIP) and the Forestry Incentives Program (FIP) • Originally authorized in 1978
Working Land assistance and incentives using payments and cost share 1-10 years
Farmland Protection Hydrology and Water Resources
• introduced in 1996 as the Farmland Protection Program (FPP) • reauthorized in the Farm Security and Rural Investment Act of 2002 (Farm Bill) as FRPP
Grassland Reserve • program started in 2003 under NRCS. Program (GRP) • 2002 Farm Bill Authorized this program from the 1985 Food Security Act
PL566 Small • authorized by the Watershed Protection and Flood Watershed Program Prevention Act PL 83-566, August 4, 1954
Greatest wetland functions and values
•
Vegetation
ÖRemote Sensing
Optimize wildlife habitat for migratory birds and wetlands dependent wildlife
•
Wetland Quantity
ÖPRS Wetlands Acreage, NWI, NRI
•
Wildlife habitat
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
•
Water Quality
ÖLake Clarity Remote Sensing, In-Situ Data
Control soil erosion
•
Soil erosion
ÖRUSLE, HEL Analysis, Tillage Analysis, LQI
Enhance wildlife habitat
•
Wildlife habitat
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
•
Buffer Strips Windbreaks, fencing and borders
• Conservation Covers and Erosion Control Management • Irrigation Water Management and Ground Water Protection • Filter and Buffer Strips • Pest Management and Nutrient Management
Improve the quality of water
• 3 focus areas: Lake Macatawa, River Raisin, Saginaw Bay
ÖFish & Wildlife Data
•
•
Establishment and improvement of riparian and aquatic areas
ÖLake Clarity Remote Sensing, In-Situ Data
•
Establishment and improvement of upland wildlife habitat, such as native prairie
• Upland Wildlife Habitat
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
• low level of implementation in Michigan •
•
Establishment and improvement of fish and wildlife habitat
• Fish and wildlife habitat • Water Quality
•
Conserve ground and surface water resources Promote conservation of the habitat of at-risk species
• Ground and surface water resources • Wildlife habitat
Ö???
• • •
Reduce non-point source pollution and groundwater contamination Reduce soil erosion and sedimentation from unacceptable levels on agricultural land
ÖLake Clarity Remote Sensing, In-Situ Data ÖRUSLE, HEL Analysis, Tillage Analysis, LQI
Incomplete program data
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
• •
Water Quality Soil Erosion
•
Air Quality
ÖEPA data, MISR, Carbon Sequestration, Odor Complaints
•
Crops Covers and Rotations
•
•
Water Quality
ÖLake Clarity Remote Sensing, In-Situ Data
•
Nutrient and Pesticide Management
• Soil Erosion and wildlife habitat
ÖRUSLE, HEL Analysis, Tillage Analysis, LQI, SVAP, Tech 12
•
Filterstrips, grassed waterways and terraces
• Create powerful incentives for other producers to meet those same standards of conservation performance on their operations • Identify and reward those farmers and ranchers meeting the very highest standards of conservation and environmental management on their operations
Forest Health and Protection
•
•
Water Quality
ÖLake Clarity Remote Sensing, In-Situ Data
•
Not same as FIP
•
Forested Landcover
ÖRemote Sensing for Forests
•
No funding after 2003
•
Working Land assistance using cost share 10+ years
•
Financial
•
•
Technical
•
Educational
• Water Quality Improvement & Watershed Protection • Afforestation, Reforestation & Wildfire Rehabilitation •
Working Land
Maintain and Enhance Natural Resourses
Improve the health and productivity of non-industrial private forestlands
Ö
Increase the future supply of timber.
•
•
Financial
• Match Conservation Program Easements (No conservation Practices)
Farmland Protection • using easements & cost share 10-30 year or permanent
Financial
• Grazing Management & Fencing
•
Financial
•
•
Technical
• Municipal & Industrial Water; & Groundwater • Recharge
•
Credit
• Watershed Protection; and Agricultural and Non-agricultural Water Management
•
Farmed Landcover
• Enhancment of plant and animal biodiversity
•
Grassed Landcover
•
• Wildlife Habitat
Protect working agricultural land from conversion to non-agricultural uses
Provide support for working grazing operations
• 6 Watersheds in 2006
• No longer an active program •
Incomplete program data
ÖCensus of Ag | Remote Sensing
•
7,000 to 10,000 acres total
ÖRemote Sensing for Grasslands
•
7,000 acres
• Enhance other forest resources. • Provide cost-effective forest improvement practices. • Continue to sustain yields and the multipurpose management of Non-Industrial Private Forest Land (NIPF).
Working land assistance using cost share • The 1996 appropriations act combined the previously 2-10 years separate program activities into a single program entitled the Watershed Surveys and Planning program.
• 2,500 to 3,500 acres per year
• Reduce emissions that contribute to air quality impairment violations of National Ambient Air Quality Standards
Working Land using base, cost share, maintenance, & enhanced payments 5-10 years
Farmland Protection using permanent easements
Michigan Implementation & Data Quality •
• Habitait, Wildlife, and Forest Health Management
• Ended in 2002 • FIP was managed by FSA prior to 1997.
Farm & Ranch Lands Protection Program (FRPP)
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
•
•
Financial
ÖRUSLE, HEL Analysis, Tillage Analysis, LQI
Wildlife habitat
•
Water Quality Buffers
Financial
Technical
Soil erosion
•
Wetland Protection & Restoration
Technical
•
•
Upland Bird Buffers
•
Financial
Measurable Source ÖLake Clarity Remote Sensing, In-Situ Data
•
•
•
•
Measurables Water Quality
•
• Native tree and grass planting · Aquatic practices Environmental • authorized in 1996 Farm Bill Quality Incentives Program (EQIP) • reauthorized in 2002 Farm Bill
Program Goals
ÖNWI, NRI, Indicator Species, MNFI, Connectivity, Simpson's
• Protection of grassland and land containing shrubs and forbs under threat of conversion to cropping, urban development, and other activities that threaten grassland resources.
Flood Prevention
•
•
Prevent damage from erosion, floodwater and sediment
•
Further the conservation and proper utilization of land
• Ground and surface water resources
ÖAquifer and reservoir level monitoring
Further the conservation, development, utilization, and disposal of water
•
ÖRemote Sensing for Flooded Land
Water Quality
Flooded landcover
Measurable
ÖLake Clarity Remote Sensing, In-Situ Data
• 25 watersheds back to 1960s
Source of measurements 18
Components of the EQI EQI = Soil + condition index
Stream + Land habitat + health index index
Societal utility index
Total soil saved – RUSLE (tons)
Stream Wetlands Buffers – created – PRS PRS (feet) (acres)
Economic data – Census of Agriculture
Odor complaints
HEL analysis (% treated)
Lake Clarity – Remote sensing
Wildlife habitat (acres)
Crop type – Remote sensing (MODIS)
Particulates – Remote sensing (MISR)
Residue cover/tillage (%)
In-situ measures
Biodiversity Index (e.g., Simpson’s)
20
20
20
20
Max. 100 = 20
+
+
Species counts
+
+ Air quality index
+
EPA air quality data
Need not have equal weights for the components. 19
Model Approach for Metric or Index Development
Measurables Analysis Reduced set of calibrated metrics Weights Assigned
Category Metric
Component 1
Component 2
Component 3…
Soil Condition
Stream Health
Land Habitat
d11 d12 … d1J
d21 d22 … d2K
d31 d32 … d3L
f11 … f1J’
f21 … f2K’
f31 … f3L’
m11 … m1J’
m21 … m2K’
m31 … m3L’
W1 w11 … w1J’
W2 w21 … w2K’
W3 w31 … w3L’
M 1 = ∑ w1 j m1 j
M 2 = ∑ wk m2 k
M 3 = ∑ w3l m3l
j
k
Overall metric or index M = W 1 M 1 + W 2 M
2
l
+ W 3 M 3 ... + W n M
n 20
Observations from Previous MTRI Metric Development Start with the interrelated input measures expressed in their own, “natural” units, divided into categories (e.g., soil condition, societal utility, etc.)
Find/develop output metrics expressed in a common, normalized unit – only calibrated within category – The relationship between the normalized metrics, m, and the input measures, d, can in general be expressed by a function, fi – In general, fi could be nonlinear and the measures, di, could be stochastic in nature – One reasonable first approximation is to use a Taylor series expansion of fi that leads to a linear relationship between the ms and the ds
21
Important Implementation Issues with This Approach How to choose, and who chooses, the “calibration” functions – Option 1 (default/automatic): choose so that the mean and spread of the resultant metrics are approximately equal • Over all of the units of interest (e.g., counties)
– Option 2: choose based on statistical analysis with input/guidance of expert opinions/analysis • Possibly use a Cost Valuation Analysis • Updated on a yearly basis (as we have discussed for the EQI itself)
How to set, and who sets, the weights – Analyst provides first nominal set of weights • Derived from program goals
– Analyst can apply different weights for what-if and sensitivity analyses – Analyst obtain guidance from expert opinion (NRCS staff, for example) • Updated on a yearly basis (in a later discussion)
22
Important Measurement Issues Important to have confidence bounds, such as shown below EQI C1 ( | ) CBound Value CBound
EQI C3 ([ | ) | EQI C2
]
– In above example, • EQI C1 is significantly below both EQI C2and C3 • EQI C3 appears to be higher than EQI C2, but need better data to help tighten up the confidence bounds
Generating a metric or index is only half of the story: also need to know the preciseness of that metric – Some resulting EQI values will be more accurate than others, depending on • Availability of data (e.g., air quality data not uniformly present or good across Michigan) • Consensus (or lack thereof) of experts in setting weights
– Aids in illuminating where there is a lack of knowledge or data and where more study/work/consensus is needed to firm up results
23
Example Data Inputs Being Pursued for EQI Data Sources Air quality data from EPA and elsewhere – Examine approaches for geographic averaging of the available data from existing monitoring stations (about 35 across the state) – Add uncertainty/sensitivity analysis to account for geographic averaging
% HEL treated (part of soil condition component) – GIS analysis by MTRI team using SSURGO and land cover map
Biodiversity measures (such as Simpson’s indices, part of habitat component)
Correlates with in-situ measurements (or literature) – Document that some practice is well correlated with positive outcomes; apply the correlate statewide wherever that practice is made
Water quality via remote sensing (reflectivity – USGS efforts)
24
Analysis of Highly Erodible Land (HEL) Purpose: Produce a map of HEL agricultural land for the State of Michigan
25
HEL Analysis with IFMAP Land Cover Lenawee County
Area (sq km)
Area as a % of Total
Ag HEL
95
6.79%
Ag Potentially HEL
332
23.73%
Ag Not HEL
973
69.55%
Ag Total
1399
26
NRCS Erosion Reduction Trends in Michigan Large drop in HEL treated (54.3%) from 2001 to 2002 900,000 772,660
Acres Treated or Tons Saved
800,000 700,000
629,964 582,726
600,000
572,338
500,000 400,000 300,000 200,000 107,224
100,000
136,489
114,985
95,292
43,627
42,529
2000
2001
19,440
19,769
2002
2003
0 Year Total Acres Treated
Highly Erodible Land Treated (acres)
Total Soil Saved (tons) 27
(CTIC)
28
(CTIC)
29
(CTIC)
30
31
Attributes of the EQI Product Enables comparisons of normalized information – Between counties/watersheds – Over time (year-to-year)
Can be displayed in tabular or map form Can be used for “what-if” and sensitivity analyses Provides a new tool for informed program management – Offers user friendly and interpretable measure of how well NRCS programs are improving environmental quality in Michigan – Facilitates funding allocation decisions – Assesses effectiveness of NRCS programs and practices
32
Next Steps in Developing EQI for Michigan Stratify counties/watersheds to minimize effects of confounding influences and examine outcome variables of interest within strata – First-order confirmation that confounding influences are not the only cause of differential outcomes—early look confirms this
Survey, formally or informally, NRCS and other conservation experts in Michigan to establish appropriate weights for indicators of EQI components and for EQI components (wis and WIs)
Complete data collection (or at least preliminary data collection) to allow for test run using recent data (e.g., for 2004 and 2005)
Complete sensitivity analyses to characterize stability of EQI – Includes characterization of uncertainty and confidence intervals
Create visualization tool (map display) 33
Conclusions MI-NRCS is interested in obtaining a metric or index such as EQI, with caveat that it is normalized and calibrated
Have established appropriate mathematical procedure for reducing multiple, disparate indicators into distinct components and overall metric
Pursuing best data to allow for statewide comparisons in Michigan by county and/or watershed – Both units of analysis have advantages
Will examine validity through review of output by MI-NRCS experts – Does EQI produce results that they believe matches ground truth?
Will link EQI to data visualization (IMS) and data management (TATS) tools also being developed in this program
34
35
