MEASURE Evaluation - Assessment of ORS Commercialization Interventions
Assessment of ORS Commercialization Interventions
Public/Private Partnerships for Child Survival
Abstract
Name: Assessment of ORS Commercialization Interventions
Purpose: The Assessment of ORS Commercialization Interventions(AOCI), a multi-study tool, was originally designed for the purposes of evaluating a multi-sectorial public-private partnership for the commercialization of oral rehydrat
ion salts (ORS) in Bolivia.
The tool measures knowledge, perception and behavioral data among consumers,
pharmacists and doctors.
Type of Design: Descriptive qualitative studies and cross-sectional household studies.
Name: Assessment of ORS Commercialization Interventions
Origin: Sandhya Rao, Camille Saadé, Luis Alberto Quiroga, Marcela Revollo
Source: BASICS
Basic Description: A quantitative and qualitative multi-study methodology to assess and
evaluate ORS commercialization interventions. The five parts of the tool include: 1) a
knowledge, attitudes and practices (KAP) household survey with mothers of children under five
years; 2) focus group discussions with consumers; 3) a simulated purchase study (SPS) in
pharmacies; 4) in-depth interviews with doctors and pharmacists; and 5) a partnership process
evaluation with members of an inter-agency task force consisting of international donors,
Ministry of Health officials, pharmaceutical companies, advertising agencies and project
managers.
Country Applications: Bolivia
Technical Scope: Child survival social marketing.
Purpose: a multi-study tool, was originally designed for the purposes of evaluating a multi-sectorial public-private partnership for the commercialization of oral rehydration salts (ORS) in
Bolivia. The tool measures knowledge, perception and behavioral data among consumers,
pharmacists and doctors.
Type of Methods: Qualitative and Quantitative.
Design: Descriptive qualitative studies and cross-sectional household studies.
Frequency of Administration: Every five years for evaluation is suggested; more frequently for
assessment purposes.
Key Users of Information
In Bolivia, data was used by members of an inter-institutional task force including donors, MOH,
project staff, pharmaceutical companies or other private sector institutions and advertising
agencies.
Objectives and Scope of Tool
Impact: To measure change in knowledge, attitudes and practices of target groups (i.e.,
consumers, pharmacists, doctors, retailers) regarding diarrhea treatment.
Assessment: To gather in-depth data on the knowledge, perception and attitude of target
groups toward generic and commercialized ORS; to determine appropriate messages and
effective channels of promotion for each target group.
Process: To gather information on the potential sustainability, replicability and
effectiveness of the ORS commercialization strategy; to provide an ecological context
(social/financial/operational) to the implementation phase of the initiative.
Key Monitoring Needs and Evaluation Questions Tool Seeks to Address
How has the introduction of a commercialized ORS affected the market demand for
ORS?
How is the new commercialized ORS perceived by the consumer?
How has the promotional campaign (mass media, interpersonal communication and
point-of-purchase materials) affected consumers' knowledge and correct use of ORS and
other treatments?
What is the effect of a promotional strategy and campaign on ORS recommendations by
pharmacists?
Which aspects of the promotional campaign most influenced the target audiences? How
was the promotional campaign perceived by the audiences?
How were prescriptions by doctors for children with diarrhea affected by the introduction
of an inexpensive, branded ORS?
How did consumer knowledge and use of non-ORS treatments for diarrhea change after
the introduction and promotion of inexpensive, branded ORS?
What are the key variables of the strategy that contribute to its sustainability, replicability
and effectiveness, according to stakeholders?
Key Indicators
Knowledge of branded and unbranded oral rehydration salts.
Source of ORS knowledge.
Perception of ORS.
Attitude toward the use of ORS.
Recommendations of ORS by pharmacists to consumers.
Prescriptions of ORS by doctors.
Use of ORS among consumers.
Where ORS is obtained by consumers.
Knowledge of ORS preparation.
Perception of promotional materials.
Research Design
Standard protocol:
Household KAP survey with mothers of children under five: A systematic sample of
respondents is taken from urban and rural areas. The sample should be representative of
the area of intervention (district/region/nation). Topics include: socio-demographic data
on age, income level, education; knowledge and use of branded and unbranded ORS;
exposure to promotional campaign messages and print materials; knowledge and
demonstration of ORS preparation.
Focus group discussions with consumers: FGDs are conducted in primary
regions/districts of interest that contain mothers of children under five who have 1) been
exposed to the promotional campaign; and 2) have used the branded ORS. The FGDs are
stratified by socioeconomic level of the participants. Topics include: knowledge of and
attitude toward ORS and management of diarrhea; perceived benefits and constraints of
branded and unbranded ORS; perception of content and presentation of branded product;
perception of promotional campaign.
Simulated purchase study: A random sample of pharmacies is taken from a list in each
urban/rural area to be sampled (lists may not be available for rural areas). The
pharmacies are visited by "mystery clients" who pretend to have a child less than two
years of age with simple diarrhea for 24 hours. The interviewers (mystery clients) then
ask the pharmacist, or person attending them, for treatment recommendations. After
leaving the pharmacy, interviewers record notes of their interaction on a standard form.
Topics include: the order and type of treatment recommendation; pharmacists'
knowledge of branded/unbranded ORS and diarrhea; recommendations on preparation of
ORS; availability of branded/unbranded ORS in pharmacies; existence and location of
point-of-purchase materials.
In-depth interviews: Interviews are held with a random sample of doctors and
pharmacists. Topics include: knowledge of diarrhea case management, including when to
recommend ORS and how to correctly prepare the solution; characteristics of branded
ORS; perception of patient demand; perception of promotional materials.
Process Evaluation: Interviews are held with a convenience sample of task force partners
(stakeholders). Topics relate to stakeholders' perceptions of strategy, and implementation
of the intervention, and the documenting of lessons learned. The purpose of the process
evaluation is to understand how operational effectiveness may be improved to ensure
sustained impact of the intervention.
Lessons from experience:
KAP study should be conducted on a sample of all socioeconomic levels such that
differential impact on various socioeconomic groups can be assessed.
KAP study should use standard format for public health impact questions (i.e.,
WHO/CDD Household Study, UNICEF MICS survey, DHS).
Market research, epidemiological analysis and evaluation objectives should be clearly
delineated prior to questionnaire development and selection of study design such that
objectives can be balanced in the same questionnaire if resources are limited. If
necessary, oversampling or additional sampling of certain areas may be considered in
order to fulfill different objectives.
In countries with many linguistic and ethnic groups, attention should be paid to collecting
a representative sample of all groups, and translation of questionnaires into the
appropriate languages.
Training
Standard protocol: All interviewers must be able to speak appropriate local languages, even if the questionnaire
cannot be translated due to time or resource constraints. Interviewers for the simulated purchase
study must resemble members of the target population, usually women of lower socioeconomic
levels. Interviewers for in-depth interviews with stakeholders, doctors and pharmacists require a
greater level of comfort with broader subject areas and ease with probing for deeper answers in
situations of asymmetrical information (i.e., when one party, usually the interviewee, has more
information about the subject area being researched.) Focus groups require moderators to
facilitate and guide discussions as well as observers to take notes and operate audio-visual
equipment. Preferably, moderators must have a higher level of knowledge of the subject area
than focus group participants, and must be able to encourage all participants to speak during the
sessions. Interviewers should be able to speak the requisite languages and/or dialects of selected
respondents.
Lessons from experience:
Interviewers should be made aware of the research objectives of each study and the
overall purpose of the multi-study research.
Interviewers and local researchers should be keenly aware of potential biases that may
occur (and how to resolve them) during questionnaire design and implementation.
Simulated purchase study interviewers should be trained well using tricks and hints to
better memorize discussions that take place in pharmacies. Exploration into the use of
observational techniques (i.e., having a second interviewer enter a pharmacy
simultaneous to the principle interviewer and observe the interaction between he/she and
the pharmacist).
All questionnaires should be pretested with respondent cultural and/or linguistic groups.
Implementation
Standard protocol:
Several studies can be undertaken simultaneously if financial and human resources are adequate.
The length of time needed for each study will depend on local logistics and resources. The
Bolivia studies were implemented over a period of four months, including study and
questionnaire design, sampling, pretesting, obtaining stakeholder approval and comments, data
collection, analysis, report writing and presentation.
Lessons from experience:
Baseline and follow-up KAP studies should be undertaken during the same season to
account for varying levels of diarrheal prevalence.
SPS interviewers should vary the route they take in selecting pharmacies to avoid
entering more than one pharmacy within the small area in the same day.
SPS interviewers should be discreet and not give away their covert role by visibly
displaying their trade tools (i.e., they should not carry tape recorders, display clipbboards
or questionnaires to pharmacists, take notes in the immediate vicinity of the pharmacy or
enter the same pharmacy more than once).
FGD participants should be chosen based on socioeconomic level. Preferably, none of
the participants should know each other. Participants should all have a similar level of
knowledge of the topic area.
Analysis
Standard protocol: Quantitative data is analyzed using SPSS software. Questionnaires are coded prior to data
collection and coding structures are created while data collection takes place.
Lessons from experience:
Database structure and coding of questionnaires (questions and responses) should be done
prior to implementation of studies.
Skip patterns should be carefully checked with questionnaires prior to implementation
and taken into account in the analysis phase.
Dummy tables should be created prior to data cleaning and analysis. If possible, graphics
should be developed and selected for presentation prior to analysis.
Research hypotheses and objectives should be clearly stated at the outset. Objectives and
hypotheses should be prioritized such that it is clear which objectives may be
compromised if necessary.
A report outline should be prepared at the beginning of the study.
Reporting
Standard protocol: Each study should have a separate report with detailed analysis of the research hypotheses and an
overall summary of all results of the multi-study tool. Summary results should be reported in an
integrated manner, categorized by topic area and reporting both qualitative and quantitative data.
Lessons from experience:
Quantitative data should be prepared using tables in the final report and graphics for the
presentation.
Qualitative narratives should follow topic or content areas rather than question order.
Questionnaires and interview/focus group guidelines should be attached as an appendix to
all reports.
Prior to implementation, thought should be given to the exact purpose and nature of
reporting, the different types of interest groups who will read the report, and translation of
the report into English or other relevant languages.
Dissemination of Results
Standard protocol: Results should be presented to all stakeholders and interested parties both locally and at the
international level. The presentation should integrate key findings from all five studies including
any statistical comparisons made to baseline studies. Study designs, methodologies and analysis
procedures should be included in the presentation.
Lessons from experience:
All audience participants should be given a summary report of results prior to attending
the presentation. The summary report outlines study design, methodology, analysis and a
brief review of key findings.
Those researchers who are most intimately involved with the analysis of the data should
be present at all presentations if possible.
Presentations should always be made with relevant stakeholders allowing for sufficient
question and answer time at the end of results reporting.
Manuals and Guidelines
Currently questionnaires, baseline and final reports are available for the Bolivia AOCI. These
reports have not been developed into standardized guidelines as yet, since field tests in other
countries have not been planned. If additional testing takes place, it will be possible to
standardize questionnaires and guidelines for implementation and analysis. Almost all topic
areas are common for ORS interventions throughout the world.
Contact Person
Sandhya Rao
BASICS Project
1600 Wilson Blvd.
Arlington, VA 22209
703-528-7474 sandhya_rao@jsi.com
