If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address for correspondence: Elena Serrano, PhD, Department of Human Nutrition, Foods, and Exercise, 201 Wallace Annex, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061-0430; Phone: (540) 231-3464; Fax: (540) 231-3916
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State University, Blacksburg, VADepartment of Agricultural and Applied Economics, Virginia Polytechnic Institute and State University, Blacksburg, VA
The Youth Expanded Food and Nutrition Education Program (EFNEP) is one of the United States Department of Agriculture's hallmark nutrition education programs for limited-resource youth. The objective of this study was to gather opinions from experts in EFNEP and related content areas to identify costs, effects (impacts), and related instruments to develop a cost-effectiveness model (instrument) for youth EFNEP, which does not exist. A cost-effectiveness model determines the economic or financial cost of producing an impact. The findings highlight several challenges in identifying inputs through consensus and provide a roadmap for the creation of a model that can be adopted by state EFNEP coordinators.
The Expanded Food and Nutrition Education Program (EFNEP) is one of the United States Department of Agriculture's hallmark nutrition education programs. The goal of EFNEP is to assist limited resource audiences inacquiring the knowledge, skills, attitudes, and changed behavior necessary for nutritionally sound diets, and to contribute to their personal development and the improvement of the total family diet and nutritional well-being.
United States Department of Agriculture, Cooperative State Research, Education, and Extension Service. Expanded Food and Nutrition Education Program (EFNEP). http://www.csrees.usda.gov/nea/food/efnep/efnep.html. Updated November 19, 2009. Accessed July 1, 2010.
EFNEP uses a unique and effective peer-education model in which paraprofessionals of the same sociodemographic community, usually indigenous to the target population, perform health education services. Begun in the late 1960s, EFNEP operates in all 50 states and the 6 United States territories and has benefited over 25 million individuals in its history. In 2009, EFNEP was appropriated $66.15 million dollars.
Cornerstone Team. Cornerstone Report from Washington: House Appropriations Committee Approves Ag Bill. http://www.land-grant.org/reports/2009/06-19.htm. Published June 19, 2009. Accessed July 1, 2010.
EFNEP has adult and youth components. For the adult component, EFNEP is delivered as a series of 6 or more lessons. At one time, EFNEP was delivered to the homemaker in her home and to small groups of 2–4 adults. Today, adults are taught individually in their homes, in neighborhood groups, or even large groups of up to 35 adults at cooperating agencies. For the youth component, EFNEP varies from location to location and from state to state. Education is delivered in group settings through school enrichment, before- or after-school programs, day camps, community centers, and neighborhood groups. Lesson topics include nutrition, food management, food preparation, food safety, fitness, and maintaining a healthful weight. As a result, it is difficult to study the impact of youth EFNEP, which may explain why there has been only 1 published effectiveness study of youth EFNEP.
No economic evaluations of youth EFNEP have been conducted. The effectiveness of EFNEP on participants' behaviors has been relatively well-studied for adults,
Nutrition behavior change among EFNEP participants is higher at sites that are well managed and whose front-line nutrition educators value the program.
Knowledge, skills, and behavior improvements on peer educators and low-income Hispanic participants after a stage of change-based bilingual nutrition education program.
Cost-benefit analysis indicates the positive economic benefits of the Expanded Food and Nutrition Education Program related to chronic disease prevention.
The purpose of this study was to gather and document opinions from experts in EFNEP, economics, nutrition, physical activity, behavioral interventions, and evaluation research in order to create a cost-effectiveness model and accompanying online software. These tools would ideally enable state- and national-level Extension faculty, administrators, and researchers to calculate and communicate the cost-effectiveness of their respective youth-targeted nutrition education programs as they relate to overweight and obesity prevention.
Background Information: Why Cost-effectiveness?
A cost-effectiveness analysis (CEA) calculates the cost of achieving some measureable effect. In this study, investigators sought to determine the economic or financial cost of producing an impact as a result of participation in youth EFNEP. The key formula in a CEA is called the incremental cost-effectiveness ratio (ICER). ICER(p) = (C1(p) – C2(p))/(E1(p) – E2(p)), where p denotes the program, C(p)i is the cost of the programs 1 (treatment or new intervention) and 2 (control or “old” intervention), and E(p)i is the effect (impacts) of programs 1 and 2. In general, the ICER is used to compare the cost and effectiveness of new and old(er) devices, techniques, drugs, or interventions. Since this study will not compare youth EFNEP programs to a control or other program, the cost-effectiveness ratio (CER) is defined as CER(p) = C(p)/E(p).
Cost-effectiveness analyses differ from cost-benefit analyses (CBA). Existing studies for the adult EFNEP programs focus more on CBA.
Cost-benefit analysis indicates the positive economic benefits of the Expanded Food and Nutrition Education Program related to chronic disease prevention.
The CBA is NB(p) = B(p) – C(p), where p, again, denotes the program; NB(p) net benefit of the program; B(p) total benefits; and C(p) total costs, all measured in dollars. Benefits include both direct benefits, such as increases in life expectancy, and indirect benefits, such as increased productivity at work. Correspondingly, the CBA conversion requires a multitude of decisions and assumptions, such as: diseases avoided by following an optimal diet; incidence of the disease/condition that is attributable to diet; optimal nutritional behavior; and cost of disease/condition avoided. As a result, the Panel on Cost-Effectiveness in Health and Medicine highly recommends the CEA:[The] health sector has traditionally favored economic analyses that assess cost per unit of health effect, resisting the use of the closely associated technique of cost benefit analysis (CBA), where both costs and benefits are measured in dollars. A number of ethical difficulties ranging from macro issues, such as the effect of valuing the time people spend pursuing medical treatment according to their wages, are already embedded in CEA. CBA adds an additional difficulty in that it presumes to put a dollar figure on the value of human life and uses controversial methods to do so. The panel has shared the dominant bias of the health sector—that monetizing the price of life in these ways introduces ethical concerns that are avoided by CEA, albeit at the sacrifice of generalizeability.
Though none of the CBA EFNEP studies considered youth, it is clear that the difficulties associated with CBAs would be exacerbated in a youth study, because of the increased uncertainties in dealing with a longer life span. As a result, it was determined that CEA would be used rather than CBA.
Methods
Participants
The expert workshop panel consisted of 12 state and national experts: 7 females, 5 males (n = 12); average age of 51.5 (n = 11); 21.3 years of related work experience (n = 11); and were predominantly white and non-Hispanic. One individual was Native American Indian, and another was Latino/Hispanic. The panel had expertise in EFNEP (including state coordinators), health economics, nutrition, physical activity, behavioral interventions, and evaluation research. All participants attended an expert panel workshop in Blacksburg, VA, on May 11, 2009 and May 12, 2009. All participants provided informed consent, and the proposed research was approved by the Virginia Polytechnic Institute and State University Institutional Review Board.
Workshop Overview
Each workshop participant received meeting materials 2 weeks prior to the workshop. The materials contained background information on relevant topics (EFNEP, cost-effectiveness ratio), biosketches of all workshop participants, and workshop objectives. The 2 main objectives of the workshop (see Figure) were:
•
Conceptualization: Determine (based on consensus) the conceptual constructs that need to be measured in terms of the costs of the program and the effects (impacts) of the program based on justified arguments and with an emphasis on overweight/obesity prevention.
•
Instrumentation: Determine (based on consensus) the instruments that can be used in measuring the conceptual constructs for costs and effects (impacts) of the program based on justified arguments.
FigureInput components of Cost Effective Ratio (CER).
The first day was devoted to content with keynote presentations on: EFNEP; youth EFNEP evaluation; cost-effectiveness; and the Virginia youth EFNEP curriculum proposed, as a case study, to be used to gather data for the creation of a model. The second day focused on discussion of the conceptualization and instruments for determining costs, followed by conceptualization and instruments for determining effects.
Analysis
A stenographer (certified court reporter) transcribed each panelist's comments and nonverbal communication (nodding, shaking of head). The workshop was also tape recorded using 2 digital audiorecorders placed on separate tables. Two panelists took notes during the meeting to document major viewpoints and decisions, affirmed by both verbal and nonverbal communication. The stenographer's transcripts were converted directly into 2 separate Microsoft Word (version 2007, Microsoft Corporation, Redmond, WA, 2007) documents for each day of the workshop (126 and 198 pages, respectively) from .txt files. Where the stenographer indicated that comments were inaudible, 2 members of the expert panel compared the transcripts to the audiorecording and workshop notes for completeness. Required clarifications were discussed between the 2 panelists until consensus was reached, with any missing information added and highlighted in the transcript. The 2 panelists then independently reviewed the complete transcripts manually, using the constant comparative analytic framework outlined by Krueger.
Data were grouped together by discussion points and themes, identified by frequency and extensiveness of their respective discussions compared to other topics, and placed in categories. Descriptive summaries of the points and counterpoints, including extracted quotes, and the final decision made by group consensus were also generated. The 2 reviewers then compared, contrasted, and discussed their respective findings to generate a final report. Selected quotes that best described the viewpoints were chosen; they are highlighted in italics in the results. All members of the expert panel reviewed the final results and discussion to ensure that they fully represented the panel's opinions and the context of the discussion and recommendations.
Results
General
Workshop attendees felt it necessary to explicity define the target audience for any cost-effectiveness model that was developed. In this regard, a number of EFNEP stakeholders were identified, including state and national legislators. However, workshop participants indicated that the key target audience for the cost-effectiveness model would be state EFNEP coordinators. Consensus was that state coordinators could, in turn, share cost-effectiveness data with their key stakeholders to demonstrate success or to be used as a metric to encourage quality improvement. For the remainder of the workshop, the discussion focused on identifying costs and effects that would be appropriate and available at the state level.
Conceptualization
For the most part, consensus was easily obtained for the cost side of the CEA (Table 1, Table 2), with the exception of a few items. The central idea that emerged is to use cost categories from accounting and economics for classifying and collecting cost data. It was noted however, that in some states, data are entered at the local level by the educator teaching the classes. In other states, the data entry is done at the state level. Further, the administrative structure of EFNEP varies from state to state. For example, some have multi-county supervisors, others do not.
Table 1Identified Costs by Expert Panel for Inclusion in Youth EFNEP Cost Effectiveness Model
Table 2Controversial Discussion Points of Expert Panelists on Identifying and Collecting Cost and Effects for Youth EFNEP Cost-effectiveness Model
Topic
Panelist Quotes
Final Decision Based on Consensus
Cost
Development, adaptation of materials
Anything that you have already done, you don't count. Anything you are going to do, you count. (If not)…It limits extension to a non-development agency and only a delivering agency.
New development (production) costs will be calculated. Costs will not be calculated retroactively for programs already developed, except for printing costs.
School teachers as volunteers
Or the teacher is helping with logistics, not delivering the program but helping with logistics, right, helping the 4-H agents enter into the school, get the students ready, help collect paperwork and get those confidentiality statements. I look at those teachers in a different way. That's 2 people getting paid to do the same job.
Teacher salary will be included, as they are critical collaborators and may assist with delivering the program and maintaining classroom order.
Opportunity cost of educator
But if an agent has other duties in addition to EFNEP and she spends 30% of her time on EFNEP, there is an opportunity cost of not just her salary, but in addition to that 30% of her salary spent directly on EFNEP is the opportunity cost of the foregone benefits of the other program she is not delivering. The dollar amount of her salary attributable to the EFNEP is a dollar measure of that opportunity cost.
Opportunity cost will not be determined beyond the salary attributable to EFNEP.
Office space
Whatever is changing as a result of your intervention; so if you have to pay rent on a place specifically because you need to house your equipment and your staff, then you count that.
A percentage of space used by EFNEP will be calculated. This information should be readily available for states administering the SNAP-ED.
Travel for parent and child
The Healthy Weights for Healthy Kids (program) is not something stand-alone over in a campsite or something by itself. It is embedded within some other activity. But they are already paying for camp and the experience and there is going to be something that fills that slot anyway, so it is the same thing.
Travel cost for parents will not be included because youth EFNEP are embedded in existing programs the child would attend anyway.
Effects
Program effects
It is my understanding that there has never been a proven effective dietary intervention under a randomized controlled trial setting…So is it not that all of the benefits of these programs are really coming through the health effects? There has never been an effective dietary program for obesity….We shouldn't mention obesity at all. I think there are studies that show—definitely show behavior change…I don't see it is as our role to be measuring the long-term health but we can show evidence of behavior change and then is the cost benefit people to translate those behaviors into long-term change, if in fact we can substantiate that those changes. The problem is proving that you can be reliably achieving these intermediate outcomes with this intervention.
Given that EFNEP's mission includes “changed behavior,” the final model and software will allow for individuals to calculate cost-effectiveness based on data on knowledge, skills, attitudes, or behavior. In other words, cost-effectiveness can be calculated without behavioral data.
Different curricula and ages
How do we deal with all of these different curricula and how do we have any kind of system that can be replicated? But I think that is the impact. You can take most of our youth programs and merge them into those [EFNEP] impacts.
The final model will focus on the existing youth EFNEP impact indicators, since all curricula should be aligned with them.
Data collection
A post- rather than pretest where you don't have them faced with the test as soon as you are trying to be introducing them to how you are going to have this wonderful experience… It (retrospective posttest) was equally successful at measuring … So we became believers in the retrospective process of putting that under a microscope. What do most states do? Traditional pre/post test. They start the EFNEP with a pretest.
Traditional pre- and posttest evaluation instruments will be used to emulate standard and existing evaluation practices.
Institutional Review Board (informed consent)
I heard over and over (in state), if it is an extension program, you don't need an IRB. But the issue, if people are delivering youth EFNEP programs across the country in different states, they are already dealing with IRBs. So the only difference here is if we want to construct an instrument that would be different, then they need IRB approval, correct? But our programs don't go through IRBs, but just the data collection evaluation. Which I have always said, if there is potential for harm, it is participating in the program, not the evaluation. And the programs don't go through IRBs.
Given the variability between states and IRBs, each state will need to take ownership of IRB guidelines and expectations. The project team will post recommendations for survey instruments and consent forms.
EFNEP indicates the Expanded Food and Nutrition Education Program; IRB, Institutional Review Board; SNAP-ED, Supplemental Nutrition Assistance Program - Education.
Direct costs will be divided between adult and youth components, with the educator's percent appointment to youth EFNEP being used to calculate labor, utilities, and capital costs for that respective site (Table 1). Finally, cost data should be gathered and collected only for programs that consist of a series of 6 or more lessons, not for 1-time presentations.
A significantly greater portion of the workshop was devoted to effects (impacts) than costs. Several challenges were noted in evaluating youth EFNEP, including: the high degree of variation in programming; age-dependent curricula to accommodate a wide range of cognitive developmental levels; a wide age range (pre-K to 12th grade); and focus, including knowledge gains, skills learned, and behaviors changed. A great deal of discussion revolved around what effects could be expected from youth EFNEP (Table 2), including: What is the evidence supporting the effectiveness and efficacy of EFNEP? Does behavior change take place? What are the long-term impacts? What instruments have been tested with youth EFNEP? The panelists also discussed using Quality Adjusted Life Years as the measure of the effect. Quality Adjusted Life Years involves translating knowledge, skills, attitudes, and behavior (the actual stated goals and desired effects of the program) into number of life years saved (as mentioned earlier under CEA vs CBA). The expert panel felt that there was insufficient evidence in the literature for making this translation with any reasonable level of confidence, particularly among youth. Given the state of evidence and the “political” importance of EFNEP as a program that elicits behavior change, the panel ultimately decided that the final cost-effectiveness model should allow for the calculation of cost-effectiveness for knowledge, skills, attitudes, and behavioral effects, as these are the stated goals of the program. The state coordinator then has the flexibility to determine which effects to ultimately include in the model. Ideally, all effects would be aligned with the following youth EFNEP indicators related to nutrition and health,
along with an impact indicator on physical activity:
1.
Youth now eat a variety of food
2.
Youth increased knowledge of the essentials of human nutrition
3.
Youth increased their ability to select low-cost, nutritious food
Instrumentation
Although 1 goal of the workshop was to identify instruments to measure cost, the collection of cost data was viewed as being rather simple and straightforward and founded on basic accounting and economic principles that did not lend themselves to measurement concerns like reliability, validity, and sensitivity. No specific instruments were identified per se to collect cost data—outside of cost-sharing spreadsheets that are routinely collected for the Supplemental Nutrition Assistance Program–Education Program.
Therefore, corresponding cost instruments will be developed with flexibility to allow for differences in collection procedures across states.
There was collective agreement that measuring program effects would be the most difficult challenge in this endeavor. It was recommended that the project coordinators reach out to the broader research arena, as well as to the EFNEP/Supplemental Nutrition Assistance Program–Education youth evaluation project database,
Children & Youth Evaluation Tools and Documentation: EFNEP/SNAP-Ed Children & Youth Evaluation Tools Project. Wisconsin Nutrition Education Program Web site. http://www.uwex.edu/ces/wnep/ncyouth. Accessed July 27, 2010.
to identify appropriate instruments that are ideally reliable, valid, and sensitive (ie, ability to measure appropriate change). The panel identified several possible survey instruments to gather effects data: National Longitudinal Survey of Youth;
Reliability and validity of the Child and Adolescent Trial for Cardiovascular Health (CATCH) Food Checklist: a self-report instrument to measure fat and sodium intake by middle school students.
National Center for Chronic Disease Prevention and Health Promotion. YRBSS: 2011 Questionnaires and Item Rationale. Centers for Disease Control and Prevention Web site. http://www.cdc.gov/HealthyYouth/yrbs/questionnaire_rationale.htm. Accessed July 27, 2010.
A subset of the workshop participants will serve as an advisory panel to provide feedback on instruments that are developed.
Adoption and Implementation of Model
Although participants agreed that a cost-effectiveness model was invaluable, they also stressed the importance of the following characteristics in order to enhance its adoption:
•
Doable: “Make the assessment as easy as possible.”
•
Believable: “The personal confidence of whether or not I actually believe that is the contribution our work is making.”
•
Explainable: “I, as a state coordinator, need to be able to take this formula and be able to explain it to anybody who says, ‘How did you arrive at that number?’ I think you really need to have it not only verbally; but you have to hand it to them one, two, three, step-by-step how to explain it so that they can feel credible.”
•
Clear: “Able to answer, ‘So what? What do I get out of this?’”
•
Adoptable: “And relatively easy and relatively soon.”
A suggestion was made to target select states to adopt and implement the model, understanding that some states will be early adopters and others laggards, aligned with the Diffusion of Innovations theory.
Further, for the cost-effectiveness model to be used, it was suggested that sample evaluation instruments be posted online and accompany the final model and software for state EFNEP coordinators to use.
Cost-effectiveness Model and Sustainability of EFNEP
There was consensus that a cost-effectiveness model for youth programs was critical for supporting, advocating for, and sustaining EFNEP, as stated by 1 workshop participant: “We need to do a better job of proving this (EFNEP cost-effectiveness) value so that we build or maintain our political capital.” Additionally, the model can help identify programs that are more or less cost effective, which can, in turn, lead to productive discussions about why.
Several concerns were raised about youth EFNEP that clearly affected cost-effectiveness interpretations and overall sustainability of youth EFNEP. For example, the next generation of federal youth program reporting software should include physical activity indicators, particularly given the significance of physical activity in overall health and weight maintenance. Perhaps EFNEP should consider nationalized, standardized curricula and evaluation to help promote consistency across states. It will be critical to assess the public value of EFNEP. In addition, longitudinal and randomized control trials assessing program effectiveness should be conducted.
Discussion
Although there is a significant need for information on the cost-effectiveness of youth EFNEP, one of the barriers to providing critical data is a lack of concensus on the appropriate outcomes that should provide the basis for such an analysis. For example, is it enough to demonstrate that the programs significantly change children's nutrition knowledge or attitudes toward healthful eating at a reasonable cost? Or changed behavior? Or change objectively assessed body composition? Or show long-term health benefits? The results from this expert panel workshop helped identify inputs and outcomes needed for the development of a cost-effectiveness model and software program that can be used by EFNEP state coordinators to analyze their respective youth programs. However, several concerns were raised that were beyond the scope of the study, namely, the lack of longitudinal and randomized control trials, along with reliable, valid, and sensitive (ie, ability to measure appropriate change) survey instruments, to determine the program's overall effectiveness. As noted by one workshop participant:We would do well to remember the advice of Voltaire: ‘The perfect is the enemy of the good.’ We are trying to make progress here. We're trying to create something that would be useful out in the field. It is not going to be perfect. We are going to have to make compromises. We are going to have to make tradeoffs, so let's recognize that we have to make improvements and we are not going to end up with something perfect, but can we get something that is better than what we have now… which is nothing.
Acknowledgments
Funding was provided by United States Department of Agriculture, National Institute of Food and Agriculture, National Research Initiative, Human Nutrition and Obesity, Project 2009-55215-05074.
References
United States Department of Agriculture, Cooperative State Research, Education, and Extension Service. Expanded Food and Nutrition Education Program (EFNEP). http://www.csrees.usda.gov/nea/food/efnep/efnep.html. Updated November 19, 2009. Accessed July 1, 2010.
Cornerstone Team. Cornerstone Report from Washington: House Appropriations Committee Approves Ag Bill. http://www.land-grant.org/reports/2009/06-19.htm. Published June 19, 2009. Accessed July 1, 2010.
Nutrition behavior change among EFNEP participants is higher at sites that are well managed and whose front-line nutrition educators value the program.
Knowledge, skills, and behavior improvements on peer educators and low-income Hispanic participants after a stage of change-based bilingual nutrition education program.
Cost-benefit analysis indicates the positive economic benefits of the Expanded Food and Nutrition Education Program related to chronic disease prevention.
Children & Youth Evaluation Tools and Documentation: EFNEP/SNAP-Ed Children & Youth Evaluation Tools Project. Wisconsin Nutrition Education Program Web site. http://www.uwex.edu/ces/wnep/ncyouth. Accessed July 27, 2010.
Reliability and validity of the Child and Adolescent Trial for Cardiovascular Health (CATCH) Food Checklist: a self-report instrument to measure fat and sodium intake by middle school students.
National Center for Chronic Disease Prevention and Health Promotion. YRBSS: 2011 Questionnaires and Item Rationale. Centers for Disease Control and Prevention Web site. http://www.cdc.gov/HealthyYouth/yrbs/questionnaire_rationale.htm. Accessed July 27, 2010.
00306-4&id=gr1.jpg){kind=link}