MEASURE Evaluation - Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs
Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs
Abstract
Name: Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs
Purpose: To improve the efficiency of programs and make the best use of investments in
controlling micronutrient deficiencies by identifying the most cost-effective program
options, their costs and, coverage. The tool introduces decision-makers to the use of
economic analysis for program planning and assists analysts in estimating the costs and
effectiveness of micronutrient strategies.
Type of Design: Cross sectional
Sample Size: Each analysis or study must include at least two options and can be used
for pre-program planning or for post-hoc evaluation.
Where Used: The assessment tool was developed based on studies done in Jamaica and
Guatemala in 1992-1995 under the USAID/LACHNS project, and was used by country
teams in Indonesia, the Philippines, Peru, and S. Africa in the USAID/OMNI project.
Basic Information
Name:Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs
Origin: OMNI/JSI Project, USAID.
Basic Description: Guidelines include step-by-step explanations, forms, worksheets and
examples on how to set study objectives, design/identify program options, calculate costs,
calculate effectiveness, develop final estimates of cost-effectiveness, and interpret the results. It
is an aid to program planning and evaluation. It is currently in draft form and needs to be revised
to capture the experience of cost-effectiveness studies carried out in Indonesia, the Philippines,
Peru, and S. Africa during 1996-1998.
Country Applications:
Indonesia, to evaluate a national program for supplementing women factory workers with
iron/folate tablets.
The Philippines, to identify the most cost-effective combination of fortification and
supplementation programs for reaching a sizable proportion of the national population at
risk of vitamin A deficiency. The study was based on existing programs.
Peru, to identify the most cost-effective alternatives (fortification, supplementation or a
combination) for reaching a sizable proportion of the national population at risk of
vitamin A deficiency. The study was based on future, planned programs.
S. Africa, to determine the costs and coverage of EPI-linked vitamin A supplementation
and maize meal versus sugar fortification programs for vitamin A deficiency control.
Purpose: The overall goal is to improve the efficiency of programs and make the best use of
investments in controlling micronutrient deficiencies by identifying the most cost-effective
program options, their costs and coverage. Decision makers can use this information for
comparing alternate strategies for delivering micronutrients. Micronutrients can be delivered
through different channels such as fortification of foods, distribution of supplements and
promotion of dietary changes. The costs and effectiveness of different approaches can vary due
to population characteristics in a country or region, access to fortifiable foods, infrastructure, and
prevailing beliefs and participation rates.
Type of Methods: Quantitative. Estimates are based on a combination of existing and new data
on the costs and effectiveness of different programs.
Design: Cross sectional. The method can also be used to project changes in future costs and
effectiveness based on illustrative alternate scenarios of variables that affect cost and
effectiveness. The method includes identifying/developing program options, using secondary
data or primary data to calculate the costs of each option, calculating the effectiveness of each
option, comparing the cost-effectiveness of each option, and identifying the best option.
Program alternatives consist of existing programs, modifications of existing programs, or new
programs that have not been implemented yet. In addition to providing an empirical basis for
selecting/designing programs, the analysis develops cost estimates that can be used for
budgeting, generates information that can be used for advocacy, and can provide an assessment
of the implications of changes in technical or operational components of programs.
Frequency of Administration: Usually once for planning or evaluation.
Key Users of Information
The results are expected to aid health and food/nutrition program planners in selecting program
strategies to combat micronutrient deficiencies, and identifying ways to improve program
efficiency.
Examples of how the results of cost-effectiveness analyses were used:
Philippines - Decision to fortify wheat flour as the main strategy, and consider a targeted
supplementation strategy to reach target groups not covered through fortification.
Peru - To consider a region-specific strategy for Vitamin A deficiency control that may include
wheat flour fortification and supplementation using a campaign approach for distribution.
S.Africa - To consider a region-specific fortification strategy based on preliminary results
showing variability in maize meal intakes.
Indonesia - To decide whether to improve technical guidelines for the national program, consider
dropping the requirement of a hemoglobin test to screen recipients of iron/folate tablets, and to
include iron/folate supplementation in the national health insurance scheme for factory workers.
Guatemala - To decide whether to develop new technology to reduce losses in vitamin A during
sugar fortification.
Objectives and Scope of Tool
The objectives are:
To provide a basis for decisions about strategies for reducing micronutrient deficiencies
that takes into account the costs and effectiveness of alternative delivery systems;
To identify/develop alternate program options;
To estimate the costs of programs;
To determine the effectiveness of programs;
To compare the cost-effectiveness and other attributes of alternate single and combined
delivery strategies.
In this tool, cost-effectiveness analysis methods and examples are applied to micronutrient
programs, especially programs for iron and vitamin A deficiency control. The main
interventions or delivery systems are:
Food fortification, especially sugar/cereal flour.
Distribution of supplements, especially prenatal iron/folate and vitamin A capsules for
children and postpartum women.
Key Monitoring Needs and Evaluation Questions Tool Seeks to Address
What are the relative costs of two or more alternative strategies that reach a given target
of effectiveness?
What is the relative coverage/effectiveness of two or more strategies that cost about the
same?
What is the cost profile or pattern of costs (i.e. comparison of different activities or
program components, type of inputs, sources of funding, levels at which activities place,
etc.) of a given micronutrient program?
How can costs of a program be reduced without sacrificing effectiveness?
How can effectiveness of a program be improved without raising costs substantially?
Key Indicators
Cost per high-risk person protected from the risk of deficiency;
Cost high-risk person reached with any additional micronutrient intake;
Annual costs;
Participant costs;
Ratio of government to other costs;
Costs at each administrative level;
Costs with and without voluntary labor and donations;
Proportion of high-risk protected against risk of deficiency;
Proportion of high-risk population that received any additional micronutrient intake.
Research Design
Standard Protocol: The following steps use the example of designing a cost-effectiveness study for vitamin A
strategies:
STEP 1. Define target groups, and the nature and magnitude of vitamin A deficiency (VAD).
Identify geographic and seasonal patterns.
STEP 2. Describe behaviors of high-risk groups.Determine possible (or actual) participation in
vitamin A supplementation and fortification activities. For example, what is the utilization of
health services in areas of VAD? Potential coverage through integration with routine
immunizations delivered at facilities, through outreach and mobile clinics; campaign activities
such as National Immunization Days (NID) for polio? Can postpartum supplements be provided
through maternity services or through BCG/EPI contacts? Community-based health programs;
private/public clinics; NGOs, etc.? What are the food consumption patterns of high-risk groups,
particularly of vitamin A fortifiable staples, e.g., sugar, cereal flours, oil/margarine, dairy
products?
STEP 3. Determine the status of current and potential delivery systems for supplements and
fortification. Check convergence with VAD patterns (age groups, frequency of contact,
seasonality, geographical, etc.):
supplements: Review indicators of coverage, quality, and policies conducive to
integrating vitamin A.
fortification: status of staple foods industries, capacity, extent to which processing and
packaging are centralized, across-border influx of unfortified foods, state of technology,
product transport/storage/turnover, industry collaboration with government, precedents on
food quality regulations and track record in monitoring/enforcement.
STEP 4. Develop program alternatives to meet national vitamin A program objectives (usually
assuring adequacy in at least 80 percent of the high risk population). Exclude alternatives based
on safety (e.g., fortification levels exceeding 3,000 RE/day by women), or feasibility. Describe
the operational details of each program alternative selected. Determine annual maintenance costs
of each alternative, using a combination of 'ingredients' and 'adaptation' methods.
STEP 5. Determine the effectiveness of program alternatives in terms of incremental adequacy
gained. Calculate the cost-effectiveness of each alternative.
STEP 6. Identify how efficiency of each alternative can be improved through reducing costs
and/or increasing effectiveness. Modify operational details, and recalculate cost-effectiveness.
Conduct sensitivity analysis for variables with uncertain values.
STEP 7. Compare the program alternatives for CE and other considerations. For example, who
will bear the financial costs? Is the program equitable? Is it feasible?
NOTE: This is meant to be an iterative process even though the steps appear linear.
Lessons from experience:
Country policies and strategies should define program objectives in terms of desired coverage,
against which cost-effective options can be assessed. This is important because cost-effectiveness changes as coverage changes. Economies of scale (i.e. falling average costs) may
happen initially but as coverage extends to the most difficult-to-reach this trend may be reversed.
Furthermore, the risk characteristics of the population are likely to change as coverage rises --effectiveness in terms of mortality gains may be several-fold higher in the hard-to-reach groups.
Therefore one should not compare a highly cost-effective option (in terms of cost per year of
adequacy gained) that only achieves low overall coverage with a program that covers a larger
population, unless differences in coverage and scale are considered. It is important to state the
coverage level at which an intervention is found to be most cost-effective.
The measure of micronutrient adequacy used in many studies (intake relative to
Recommended Dietary Allowances or RDAs) is not a perfect proxy for health outcomes. This
is because: a) factors related to illness and bioavailability can modify the impact of
fortification or supplementation; b) the impact on mortality can vary among sub-groups due to
differences in severity of the deficiency at entry in the program and cause-specific mortality
levels; c) achievement of an 'adequate' level suggests a threshold effect that is not borne out
by existing studies. Further research is needed to elucidate the relationship of program
outputs and coverage with health outcomes (and in the case of iron deficiency in adults, on
productivity).
Some country teams considered adequate intake by women of reproductive age as being
equally important as intake by children. This resulted in significantly different cost-effectiveness results. Results of alternate assumptions about priority target groups should be
presented.
Cost estimates should be based on a representative sample of program sites. National
estimates derived from sample surveys should take into account variation by scale of program
operations, province/district management, infrastructure and logistics, and population
characteristics.
The careful consideration of incremental but true opportunity costs can be difficult when
alternatives such as immunization-linked versus stand-alone vitamin A supplementation
options are costed and compared.
The estimation of adequacy in vitamin A intake before and after programs requires the
estimation of individual vitamin A intakes at baseline, and modeling the incremental adequacy
gained from fortification and supplements at a highly disaggregated (preferably individual)
level. Using group averages may bias the results.
In general, country reports placed adequate emphasis on program background and policy
setting, but clear policy conclusions based on the study results required substantial discussion,
broad input from various types of experts, and adequate time to take various criteria (not only
cost-effectiveness) into account.
Future studies should take into account anticipated epidemiology trends or changes in
consumption patterns or health services utilization. They should project alternate scenarios
and revisit the cost effectiveness estimates (and coverage of high risk groups and other
decision criteria) with these projections.
It is important to clarify that the term 'costs' does not refer to budget outlays, but the value of
resources used. For example, in vitamin A supplementation programs in particular, a large
proportion of 'costs' is actually the value of existing staff or volunteer time. Financial
implications for budgets should be clearly separated. A similar concern about fortification
costs is that industry should not be assumed to carry the full additional cost burden for higher
levels of fortification. A simple calculation will show the reader that when all consumers
share these costs they are very small and very likely to be passed on to the consumer by
industry without any negative effects on demand for that product. Generally, government or
industry statistics contain data on price increases of staples over time, and a study using this
tool can show how insignificant the proposed increase in price (per kilogram) will be relative
to historical price increases.
Training
Standard protocol: Two workshops are required: one at the start of the study and one at the end. Participants are
then sufficiently experienced to conduct additional studies with limited technical assistance.
Lessons from experience: During the learning phase, it is recommended that surveyors have access to on-the-ground
technical assistance in planning, implementation, analysis and interpretation of results.
Implementation
Standard protocol: The steps include: development of program alternatives for the study based on discussions with
policy-makers and implementers; collection and assessment of existing data; decision to collect
new data; survey design and data collection analysis; developing preliminary and final cost and
effectiveness estimates; cost-effectiveness ratios calculated; other criteria compared; workshops
and discussion with decision-makers and implementers held throughout the process; and final
recommendations developed and discussed with stakeholders.
Lessons from experience: Interviewers: Two main skills groups are required. Costing data should be collected by people
with costing, financial or accounting backgrounds. For the effectiveness study, surveyors should
have experience with household surveys. If biological samples are collected, additional skills or
practitioners are required. The number varies depending upon data needs.
Cost and Financing: Studies noted above cost $30,000 per country study plus technical
assistance.
Time Requirements: Ranges from three to nine months. If secondary data on effectiveness is not
available, primary data must be collected thus extending the time required to complete the study.
Analysis
Standard Protocol: Computer Hardware and SoftwareRequirements: Spreadsheets, modeling and multivariate
analysis (e.g. SPSS).
Data Entry Techniques or Tricks: None.
Analytical Requirements: Knowledge of nutrition indicators, calculation of adequacy and
reduction in risk of micronutrient deficiencies is essential. Assessment of effectiveness and
attributing program effects requires knowledge of research design principles. Economic
principles of opportunity costs, marginal costs, discounting, etc. are required. Also analysts
should be able to project costs from a sample of program sites.
Time and Labor Requirements: At least two to three months are required to develop preliminary
estimates, fill in data gaps, conduct sensitivity tests, develop alternate program scenarios and
include concerns of the study audience in the analysis.
Reporting
Content: Reports contain documentation of the policy context and decisions to be made;
description of program options being compared and why certain options were selected; costs,
effectiveness and cost-effectiveness ratios; choice of variables for sensitivity analysis and results.
Other criteria for selecting best strategies are discussed.
Flow of Information: From analysts to implementers, decision-makers and stakeholders,
especially funders.
Presentation: Tables, bullets on recommendations; studying weaknesses or qualifications in
interpreting the results should be noted.
Lessons from experience: For decision makers it is important to put the findings in perspective. Reports should indicate
that the actual numbers are less important than relative costs and effectiveness. The process of
detailing program activities, types of inputs and target groups reached often provides insights into
improving program performance even without completing the calculations. The concepts of
cost-effectiveness comparisons, marginal costs and payoffs, and comparing
combined/incremental strategies (rather than individual interventions before a program begin)
can be difficult to comprehend. Visuals and graphics are recommended.
Dissemination of Results
Lessons from experience Significant time should be spent on ensuring that results are well understood by those affected by
the recommendations. Examples of key audiences are: industry representatives for fortification
programs, EPI program managers for EPI-linked strategies, voluntary organizations and others in
addition to government representatives (MOH, trade and industry, quality control/regulatory
bodies, agriculture) and donors who support micronutrient programs.
Manuals and Guidelines
The guidelines exist in draft manual format. They are available from USAID/G/HN, and from
John Snow Incorporated, Rosslyn, VA.
Contact Person
Tina Sanghvi
BASICS Project
1600 Wilson Blvd.
Arlington, VA 22209
703-312-6800
