Defining and measuring success is easy—if you are
Rube Goldberg. A widely acclaimed 20th century cartoonist, Goldberg
depicted outlandish inventions that accomplished simple tasks through
an intricate series of linked steps, each one triggering another
until a desired outcome was reached. Success, in Goldberg’s
world, was clearly defined and could be attributed directly to the
completion of several sequential, though highly improbable, cause-and-effect
actions. Success, in the real world, even when it is clearly defined,
is not so easily measured. Gauging the success of government programs,
in particular, can be downright complicated, even when the principles
used in designing them are rather simple.
Most conservation programs, for example, are designed to improve
the environment by offering incentive payments to farmers, who are
thereby induced to change their farming practices. Those changes
in farmers’ practices—be they reducing pesticide use,
adopting conservation tillage, or constructing a riparian buffer—should
then lead to enhanced environmental quality. But, unlike the chain
of events in a Goldberg invention, the actions involved in a conservation
program take place not in isolation, but, rather, within a larger
set of complex interactions, making it difficult to link programs
to actions to outcomes.
The first step in measuring the success of agricultural conservation
programs—and other programs designed to address agri-environmental
issues—is linking a change in farmers’ stewardship behavior
to the program being evaluated. Because many other factors (including
other government programs) influence farmers’ choices, it
is critical to determine the extent to which it was a given conservation
program incentive that stimulated some farmers to do something that
they would not otherwise have done. A second step requires assessment
of how the portion of observed stewardship behavior that can be
linked back to conservation program incentives then affects environmental
quality—given that other factors also affect the environment.
Gauging Farm Operators’ Responses to Program Incentives
Farm operators are the target of conservation program incentives,
even though the program itself aims to target one or more environmental
enhancements. Thus, to evaluate the program, one must determine
exactly how program incentives induced operators of farms of various
types, sizes, or features to “sign up” as program participants.
Then, for those who become program participants, it is important
to find out how the type and extent of conservation practices they
adopted relate to the levels of incentives provided through the
program. Only by separating the influence of program incentives
from other factors that affect farmers’ conservation choices
can the program evaluator be confident that it was the program being
evaluated that had an effect, not other circumstances.
A farmer may adopt conservation practices for a myriad of reasons.
He or she may be an ardent environmental steward who would implement
a particular practice (like maintaining grassed buffers between
cropland and water sources) regardless of program incentives. Alternatively,
a farmer may adopt an environmentally friendly practice wholly or
partly in order to increase profits. ERS research on conservation
tillage, for example, demonstrates that good
stewardship can also be good business. Policy incentives aren’t
usually required to induce a farmer to adopt what he or she views
as good business practice; market forces should do the trick in
this regard.
In evaluating the effectiveness of incentives to induce farmers
to participate in conservation programs, it is important to note
that conservation programs are not implemented in a policy vacuum.
Both the costs and benefits of participating in a given program
will vary as a direct result of the confluence with other government
programs. For example, commodity programs influence some crop prices,
making it more or less economically advantageous to manage the crops
in ways that enhance environmental quality. Input use is sometimes
controlled through quantity restrictions and use regulations. Input
prices may also be influenced by policies—including labor
laws, pesticide regulation, and subsidization of irrigation water—that
influence relative input prices and, thus, the financial costs or
benefits of conservation practices that shift input use patterns.
Finally, technological change, economy-wide variables (such as interest
rates and unemployment rates), and farm household constraints (such
as the role of off-farm work in farm household income) are also
likely to influence farmers’ decisions about farming practices—whether
or not a conservation program incentive is added to the mix.
Because farmers may adopt conservation practices for reasons unrelated
to the conservation program, simply identifying changes in farmers’
practices (let alone environmental quality) is an insufficient basis
for judging the success of a conservation program. One has
to be able to determine what proportion of farmers’ practices
can be attributed to a particular program before the success of
the program can be assessed.
Lynn Betts,
USDA/NRCS
Isolating the effects of program incentives from the
effects of other factors potentially influencing farmers’
observed conservation practices demands a lot of data of particular
sorts. A necessary requirement is the collection of data that enable
statistically reliable comparisions of farming practices by farmers
before and after program implementation, or by farmers who did and
did not participate in the program in a given year or years. Statistical
analysis of such data can support or refute a correlation between
farm practices and conservation program provisions.
However, supporting or refuting simple correlation is not sufficient
because that correlation may be spurious and because it does not
prove causality. A “before-and-after” comparison, for
example, might miss the strong influence of a new program on participants’
behavior if other factors, such as unusual weather conditions, prevented
a large number of the participants from following through on their
program-induced good intentions. Similarly, a “with and without”
comparison could falsely attribute observed conservation practices
to the conservation program if all farmer participants in the program
were pre-inclined toward voluntary environmental stewardship even
without the program, and nonparticipants were disinclined. More
information is needed than simply who participated and what practices
they employed if a strong case is to be made that the program was
the stimulus for farmers’ adoption of observed practices.
Additional data are necessary to separate the effect of a conservation
program incentive from the effects of concurrent changes in market
prices, weather, other policies, and technology. Identifying the
farmers for whom program incentives induced adoption of conservation
practices requires data on the characteristics—types and locations—of
both participating and nonparticipating farmers, the circumstances
under which they made a participation decision, the amount of the
incentive to which they did or did not respond, and regional and
other variables.
Lynn Betts, USDA/NRCS
A close look at outcomes associated with the Conservation
Compliance provision of the 1985 Food Security Act reveals the importance
of isolating the effects of the program in order to measure its
success. The provision requires agricultural producers to implement
soil conservation systems on highly erodible (HEL) cropland to remain
eligible for farm program payments. Annual soil erosion on U.S.
cropland declined by 40 percent between 1982 and 1997, suggesting
that compliance mechanisms encouraged greater conservation effort.
However, erosion also declined on cropland not subject to compliance
requirements, demonstrating that other factors must also have played
a role in reducing soil erosion. On farms for which conservation
practices could have increased net returns to farming, for example,
adoption may have eventually occurred regardless of effects on soil
erosion. In fact, after accounting for other factors, such as erodibility,
commodity program payments, and land use changes, ERS
research shows that only about 25 percent of overall erosion
reduction between 1982 and 1997 could be directly attributed to
Conservation Compliance. Even on the HEL lands targeted by the provision,
about 11 percent of erosion reduction during that period was due
to factors other than Conservation Compliance.
Linking Farmers’ Choices to Environmental Quality
Measuring changes in farmers’ practices that result
directly from conservation program changes tells only part of the
story. Conservation programs are not designed simply to induce a
change in conservation practices, but to change those practices
in order to improve water quality, air quality, wildlife habitat,
or a host of other environmental attributes. More and more frequently,
conservation programs aim to improve all of those environmental
attributes at once.
Connecting the dots that link a program’s incentives to success
in achieving that program’s environmental goal(s) is difficult
in general, but can be especially challenging when evaluating conservation
programs. Most of these programs address “nonpoint”
sources of pollution, such as the nutrients, sediments, pesticides,
and salts that enter water diffusely in runoff. In comparison to
“point” sources, such as factories and municipal plants,
which discharge through a pipe, ditch, or smokestack on which a
meter can be installed, nonpoint sources are not so easily measurable
and have an environmental effect only in the aggregate.
For example, the goal of a particular conservation program might
be to address water quality problems caused by agricultural production.
Evaluating a program based on that objective would require data
on the entire set of actions and outcomes associated with agricultural
production. Farmers control their inputs and crop production practices.
Their management decisions, including which crop is produced on
which field and with what combination of inputs, can affect water
quality, but gauging whether or not and how much they actually do
affect water quality is a difficult task. Farmers’ decisions
may lead to field-level emissions (through runoff or leaching) of
potential pollutants, such as sediments, nutrients, and chemicals,
which are difficult to monitor. Depending on the location of the
field and other physical and environmental factors, an emission
may or may not find its way to the target water body.
But even that sequence of events is only part of the story. The
last piece involves the underlying objective: What is it about water
quality that concerns us? Is the goal to reduce nutrient concentrations
in drinking water? Is it to provide improved fish habitat, perhaps
to increase recreational fishing benefits? Once a (potential) pollutant
reaches an environmental sink, such as a river or aquifer, it may
or may not have ecological or human health implications, depending
upon its toxicity, the number of other sources emitting the same
pollutant, interactions with other pollutants, and the total emissions
simultaneously reaching the environmental sink. While scientists
know much about the relationship between nitrogen runoff and tillage
practices, and the effects of nitrogen levels on biological functions,
less is known about how nitrogen is transported from a myriad of
individual fields to specific water bodies or other sinks.
In evaluating the effects of a conservation program on environmental
quality, the nonpoint source issue is compounded by the exceptional
site specificity of many agri-environmental events. Soil losses
(or other pollutants) at one location may have a different effect
on the environment than an identical level and type of soil loss
in another location. Furthermore, similar levels of environmental
effects vary in value among locations depending upon the proximity
of human populations or economic activity to the site of the damage.
For example, if a program objective is to help restore a recreational
fishery, water quality improvements that increase fish populations
closer to cities and where interest in fishing is particularly high
will be higher valued than equivalent changes in fish populations
in regions of the country that are sparsely populated or where interest
in fishing is low. Estimating monetary-equivalent values for environmental
improvement is a particularly difficult task that, while not necessary
for judging whether or not a conservation program met its goals,
is essential to determining how efficiently those goals were met.
Models Simulate What We Cannot Observe
Environmental process models can help overcome the nonpoint source
and site specificity complications of conservation program evaluation
by substituting predictions from models for direct observations
of effects. For example, site-specific changes in (in-field) soil
erosion due to particular erosion control practices can be estimated
using the Universal Soil Loss Equation and the Wind Erosion Equation.
Both models provide reasonably accurate results and require only
minimal data (a total of six variables) describing climate, topography,
soil, and cropping information at the field level. In contrast,
models of nutrient and pesticide runoff are far more complex, simulating
multiple environmental effects from the transport and fate of multiple
pollutants into environmental sinks. These “fate and transport”
models require a lot of data, often necessitating the use of dozens
of variables.
Any one process model has unique advantages and disadvantages, depending
on the indicator of interest, but relatively few are capable of
simulating the environmental effects of changes in agricultural
practices on a national scale. (See box, “Some
Agri-Environmental Process Models.”)
Some Agri-Environmental
Process Models
A myriad of agri-environmental process models exist, ranging
from simple linear calculations suitable for a handheld calculator
to extraordinarily complex computer programs requiring high-powered
machines and extensive training to operate, and from those
calibrated to a single watershed to models developed to provide
national-scale estimates. Three process models with acceptance
among a wide range of analysts include one that is particularly
comprehensive and predicts emissions at “edge of field”
and two that attempt to link practices to water quality.
USDA’s Erosion-Productivity
Impact Calculator (EPIC)—a mechanistic simulation
model used to examine long-term effects of various components
of soil erosion on crop production. The model has several
components: soil erosion, economic variables, hydrologic
conditions, weather, nutrient composition, plant growth
dynamics, and crop management.
USDA’s Soil
& Water Assessment Tool (SWAT)—a river basin
scale model developed to predict the water quality impact
of land management practices in large, complex watersheds.
Required input data include weather, soils, crops, pesticides
and nutrients.
U.S. Geological Survey’s SPAtially
Referenced Regressions On Watershed Attributes (SPARROW)—a
statistical model that relates in-stream water-quality measurements
to spatially referenced characteristics of watersheds, including
contaminant sources (such as farm fields) and factors influencing
terrestrial and stream transport.
A final complication: Model results are unlikely to match
real world observations because farming practices aren’t the
only things that affect environmental quality. Floods or drought
can damage the environment even under the very best management practices.
A given level of runoff may cause no environmental damage in a wet
year but may significantly harm fish and wildlife in a dry year
when streams have insufficient flows to dilute the runoff to nonharmful
levels. Likewise, a single watershed may well experience pollutant
discharges not only from agriculture, but also from industrial sources,
municipal water treatment plants, urban runoff, aerial deposition,
and even natural seepage. Thus, the influence of unmodeled events
needs to be extracted to reconcile simulation results with measurements
made on the ground.
Identifying Appropriate Environmental Indicators
Just what is the best indicator by which to measure environmental
quality change in the policy evaluation context? Regardless of whether
it will be measured directly or simulated with an agri-environmental
process model, the indicator(s) by which a given program will be
evaluated must be carefully selected. Reflecting broadened public
concerns, conservation programs increasingly target multiple environmental
quality goals. Along with reductions in soil erosion, potentially
measurable goals have expanded to include improved water quality
and conservation of wetlands and wildlife habitat. Newer program
objectives may include preserving open space, managing nutrients
from fertilizers and livestock waste, reducing pesticide runoff,
improving air quality, reducing greenhouse gas emissions, or sequestering
carbon in soil.
The appropriate indicator for evaluating a program’s
success must map to an aspect of environmental quality that the
program aims to address. But that’s not enough. It must also
link directly to those changes in conservation practices induced
by the program. For example, a measure of ambient downstream water
quality, such as nitrogen concentration, may appear to be an ideal
indicator of the success of a conservation program that aims to
improve water quality. But if agriculture is only a small part of
the aggregate water quality problem, ambient water quality may be
getting worse, even with a wildly successful conservation program
in place. The ambient water quality indicator may not measure the
factor of interest, which, in this example, is agriculture’s
contribution to water quality, and thus is not a good choice for
evaluating this agri-environmentally oriented program. In this case,
a less direct measure of water quality, such as pounds of nitrogen
discharged into the water body from farm fields, may actually be
a better indicator.
Appropriate indicators are:
• Policy relevant—provide a direct link to both the
environmental attributes of concern and the behavioral
changes associated with the evaluated program incentives;
• Measurable—based on sound science and make use of
data that are available or could feasibly be collected;
• Reasonably priced—cost-effective in terms of data
collection, processing, and dissemination; and,
• Easy to interpret—communicate essential information
to policymakers and other stakeholders.
Charlie Rahm, USDA/NRCS
Putting It All Together
The voluntary nature of most U.S. conservation programs, the human
factors involved in farmers’ decisions to participate (and
to what extent), the complexity of farm household decisionmaking,
and the nonpoint source and site-specific nature of agri-environmental
problems combine to make evaluation of conservation programs a data-intensive
and technically challenging process. To be successful, program evaluations
must answer both of the following questions explicitly, through
estimated, simulated, or directly measured means.
1. How do different farm operators in different circumstances
decide what production and conservation practices to implement,
in the presence and absence of the conservation program being evaluated,
at different levels of incentives provided by that program?
Isolating the unique effect of conservation program
incentives on farmers’ practices requires analysis to extract
the influence of other (policy, household, general economic, etc.)
factors that affect farm-level decisionmaking. This, in turn, requires
evaluators to collect data on the full set of factors potentially
affecting farmers’ decisions, in sufficient volume and across
diverse farm and land types and locations, to allow statistical
segregation of program-related effects from those of other influential
factors.
2. How do the farm practices attributable to conservation
program incentives affect environmental quality?
Lynn Betts, USDA/NRC
Isolating the unique effect of farm practices on environmental
quality requires program evaluators to determine where, and under
what resource conditions, practices implemented in response to the
program are located, and to designate appropriate agri-environmental
indicators for measuring program success. Process models that simulate
the complexities involved in the transport of agricultural runoff
from multiple fields to environmental sinks may help link environmental
performance with farm practices. But even then, additional analysis
is required to reconcile model predictions with real world observations.
The complicated series of cause-and-effect relationships associated
with conservation program evaluation seem beyond even the imagination
of Rube Goldberg. Many factors must be accounted for to determine
the portion of environmental enhancements directly attributable
to program incentive-induced changes in farmers’ practices.
Still, carefully designed survey and monitoring programs encompassing
each of those relationships in a coordinated fashion make such evaluation
not only feasible, but well within reach.
Download PDF version