Abstract
A protocol for a systematic onsite review of the Expanded Food and Nutrition Education Program and Supplemental Nutrition Assistance Program–Education was developed to support quality programming and ensure compliance with state guidelines and federal regulations. Onsite review of local nutrition program operations is one strategy to meet this goal. Observation and interaction with staff allow a comprehensive understanding of strengths, weaknesses, and emerging issues. This information provides managers with timely feedback to strengthen and improve all aspects of nutrition programming.
Key Words
Introduction
Quality assurance is a critical process to ensure excellence in delivery of community nutrition programs such as the Supplemental Food Assistance Program–Education (SNAP-Ed) and the Expanded Food and Nutrition Education Program (EFNEP).
1
In New York State (NYS), these programs are funded by the United States Department of Agriculture (USDA) and delivered through local county Cooperative Extension associations. Faculty and staff in the Division of Nutritional Sciences at Cornell University have the responsibility of providing statewide leadership, including monitoring programs and providing technical assistance aimed at ensuring quality. Tools and processes for these activities are not predetermined by the USDA but are at the discretion of each state leadership team. This article describes the protocol for a systematic onsite review of local EFNEP and SNAP-Ed programs developed in NYS to support quality programming and ensure compliance with state guidelines and federal regulations.EFNEP serves low-income families with children.
2
SNAP-Ed serves individuals receiving or eligible for SNAP benefits, ie, at 130% or less of the federal poverty level.US Department of Agriculture. Expanded Food and Nutrition Education Program Policies. http://www.csrees.usda.gov/nea/food/efnep/pdf/program-policy.pdf. Published October 1983. Accessed June 28, 2011.
3
In NYS, 34 county extension offices have EFNEP and SNAP-Ed; an additional 23 offices have SNAP-Ed only. Education is delivered by front-line, paraprofessional educators in both EFNEP and SNAP-Ed. These staff are supervised and the program is managed by a professional extension educator with degrees in nutrition, public health, health education, or family and consumer sciences. A county executive director is responsible for all programs in a local Extension office. The 57 counties with EFNEP or SNAP-Ed are divided into 7 geographic regions, each with a regional coordinator who is a master’s level nutrition professional. Regional coordinators provide training and facilitate communication for their group of counties.Food and Nutrition Service, US Department of Agriculture. Food Stamp Nutrition Education Guiding Principles. 2009. http://www.fns.usda.gov/ora/menu/FSNE/GuidingPrinciples.pdf.Published, 2011 Published 2009. Accessed June 28, 2011.
Nutrition education is delivered in a similar fashion in both programs. Research-based curricula are reviewed by state leadership, and those deemed of sufficient quality are available for use in the NYS programs. Annually, county managers choose among these curricula according to the perceived needs of the population they are able to reach and the interest of staff and commit to using these for the year. A minimum series of 6 lessons is delivered in one on one (usually in participant’s home in rural areas) or in small groups (3-15 participants). Enrollment procedures and records are maintained in a similar manner for both programs. Evaluation data are collected pre- and posteducation and include demographics, a self-reported behavior checklist, and a 24-hour recall. The behavior checklist includes 10 items federally mandated for EFNEP and an additional 8 to 11 items chosen from the national Nutrition Education Evaluation Reporting System–5 (USDA, Washington, DC, 2008) to assess the behavioral constructs identified in the curriculum being delivered, ie, the additional items are matched to the curriculum. Evaluation data are submitted electronically to the state office with the County Reporting System–5, designed to collect local data and feed them into the Nutrition Education Evaluation Reporting System–5.
Nutritional sciences faculty and staff provide nutrition leadership for both programs, as well as administrative oversight for EFNEP. Administrative oversight for SNAP-Ed is provided by the NYS Office of Temporary and Disability Assistance, the state agency that administers SNAP. Nutrition program policies and procedures, educational strategies and materials, staff training, professional development, and technical assistance are coordinated by a team of Cornell campus–based staff in the Food and Nutrition Education in Communities unit. Ensuring that local strategies to achieve goals are appropriate and progress is being made toward output and outcome goals is the responsibility of this unit.
A variety of methods is used to monitor program operations. Electronic submission of pre- and posteducation evaluation data by county staff is required quarterly and is systematically reviewed by campus staff. A feedback process that includes e-mail communication to county data managers is in place to clarify questions when data do not fit expected patterns of quality (eg, number of participants enrolled per full-time equivalent staff). However, monitoring county data is an incomplete assessment of program operations. Achieving a high level of confidence that local programs are operating as expected requires onsite observation of staff in action within the context in which they work.
An onsite visit by campus staff is an opportunity to emphasize the importance of nutrition programming, demonstrate support for nutrition supervisors and paraprofessional educators, strengthen communication between campus management staff and individual nutrition educators, strengthen communication among programs within regions, reinforce best practices, assess uniform application of policies and procedures, and solicit direct input for staff development. To this end, a protocol for review of program operations was developed to ensure that (1) best practices are incorporated into research-based nutrition education in a manner that meets program expectations, and that (2) all federal requirements and state guidelines are met.
Quality assurance is the process of monitoring program operations against established standards of service delivery or care.
4
The accuracy of nutrition information provided, the use of research-based adult education techniques, and the nature of interactions with participants are all aspects of standards for quality program delivery. Each of the federal nutrition programs has specific definitions of target audience, nutrition education content, and delivery methods. However, procedures for ensuring program standards and quality are at the discretion of the state leadership and should be uniquely designed to address program delivery in context. State-level stewardship of federal resources must include a review of required outputs and outcomes. Application of an assessment process that highlights and reinforces quality programming and values and rewards excellence strengthens the entire nutrition education program from the local to the federal level.Quality Assurance Protocol
A standard but flexible protocol was developed for a systematic review of nutrition programs conducted by Cornell Cooperative Extension staff in 2000. The goal was for a team of campus staff to visit each county at least once every 2 years. Before this, site visits were conducted on an irregular cycle and no standard data collection or reporting protocol was used.
The development of the protocol, reporting system, and follow-up procedures was intended to serve as a means of improving nutrition program quality and identify needs for training and support. Unlike other compliance reviews, the intent is quality assurance in nutrition education, recordkeeping, and staff performance. Specifically, the team reviews program management, communication among staff at all levels, recruitment and retention of program participants, and educational strategies, including materials used, quality of group facilitation, completeness and management of data, and staff development. The process includes review criteria consistent with the “Food Stamp Nutrition Education Management Evaluation Review Guide for State Management Evaluation Reviewers.”
5
This document is used by regional Food and Nutrition Service staff to review state plans and state operations whose purpose is to “ensure that operations comply with the requirements of Nutrition Education State Plan Guidance and are consistent with the approved plan.” The NYS protocol takes into account the concerns of the federal programs for state-level management and incorporates state-specific guidelines for quality program delivery at the local level.The site visit team was made up of 2 to 4 members, depending on the size of the county program. The team included some combination of state-level staff (unit director; program coordinator; staff responsible for data management, including training of staff on data collection and reporting; and staff responsible for nutrition materials and curriculum) plus 1 regional coordinator. A regional coordinator from another region in the state usually participated to provide an outside perspective. Team members were assigned leadership for different tasks: record review, meetings, and observations. All or most of the team meets with the nutrition manager and paraprofessional educators and participates in record review.
The site visit is a day-long process that provides an opportunity for fact finding and dialogue. Information is gathered through interview, observation, and document review (Table 1). Meetings with the executive director and nutrition supervisor include an overview of the visit purpose and schedule and introduction of the team members, as well as opportunity to discuss local issues that affect nutrition programs. Individual meetings with the local Cooperative Extension executive director, nutrition supervisor, and paraprofessional educators are the most intensive portions of the site visit, requiring 2.5 to 3 hours in total. Observation of a group or individual nutrition education session takes about an hour. Review of records, including group and individual progression records, attendance records, and lesson plans, can take the team up to 2 hours.
Table 1Outline of a Typical Day
| Activity | Time |
|---|---|
| Meet with county executive director | 15 min |
| Meet with nutrition program supervisor | 15 min |
| Meet with all paraprofessional educators as a group | 1.5 h |
| Meet with nutrition supervisor/extension educator | 1 h |
| Review records | 2 h |
| Observe group/individual education | 1 h |
| Meet with nutrition supervisor/extension educator to recap findings | 1 h |
| Lunch break | 1 h |
| Total time | 8 h |
Tools
A variety of tools was developed to streamline scheduling, data collection, record review, and report writing for the site visit, including an appointment letter template, interview form, record review form, lesson observation form, and report template.
Appointment template
A standard letter for confirming the date, schedule of activities, and staff participating is sent to the nutrition supervisor. The supervisor is responsible for organizing staff, scheduling space, planning lunch, and scheduling observations of in-home or group education.
Interview form
The interview form is designed to capture information gathered as a result of discussions with nutrition education program staff. A question guide (Table 2) is used as a basis for the interviews with the nutrition supervisor and paraprofessional educators.
Table 2Question Guide by Staff Title
| Questions for Executive directors | |
| Programming | How do you feel about the way nutrition programs are operating? What do you hope we accomplish today? |
| Questions for Nutrition Supervisors | |
| Recruitment | Describe your recruitment strategies. What strategies do your paraprofessionals use to recruit participants? (What guidelines do you have for recruitment?) Describe how you recruit SNAP recipients. Describe how you recruit EFNEP participants. What is working well? What are the biggest challenges? |
| Retention | What are your biggest retention struggles? What guidelines do you have to support paraprofessionals in terminating participants? |
| Supervision of paraprofessionals | What is your plan for supervising paraprofessional work? How often do you meet with paraprofessional staff in groups, individually? How do you know what/how they are doing? What supervisory responsibilities could we help you with? What formal processes do you have in place for observation of staff with participants? How often do you conduct these observations? |
| Programming | Tell us about nutrition programming. What curricula are being used? What additional nutrition education materials are being used, eg, handouts? Do recipes have a nutritional analysis and meet state guidelines? What other nutrition resources are you or your staff using on a regular basis? |
| Data management | How do you ensure that all staff have completed the institutional review board’s online training for use of human subjects? What is the relationship between your data entry person and your paraprofessionals? What procedures or guidelines are in place to guide paraprofessionals in data collection? How comfortable are you in interpreting your program data? What strategies do you use to ensure accuracy in data collection and entry? |
| Staff development | Have all staff participated in initial paraprofessional training? Have follow-up meetings been scheduled with the supervisor after each session of training? For what topics/areas would in-service or other training be helpful to you or staff? |
| Questions for Paraprofessional Staff | |
| Recruitment | Describe how you recruit new participants to the program. What strategies are most useful? What does not work well? What are the biggest challenges? |
| Retention | What are your biggest retention struggles? What proportion of participants graduate? What guidelines do you have for follow-up when participants are “lost”? |
| Supervision | How do you know whether what you are doing is on target? When is the last time your supervisor observed you teaching a group/individual? What formal feedback have you received as a result of these observations? |
| Programming | Tell us about programming with participants. What curriculum are you using with participants? What resources do you most often use? How do you know when a participant is ready to graduate? |
| Data management | What is your relationship with your data entry person? What procedures or guidelines are in place to guide data collection? How comfortable are you in interpreting program data? What strategies do you use to ensure accuracy in data collection and entry? |
| Staff development | For what topics/areas would you like additional in-service training? How frequently are you able to participate in regional or county trainings/in-services? How have follow-up activities been scheduled? |
SNAP indicates Supplemental Food Assistance Program; EFNEP, Expanded Food and Nutrition Education Program.
a Deleting participants from the active participant list after multiple no-shows.
Record review
The purpose of the record review is to verify how well Nutrition Education Evaluation Reporting System–5 data elements are documented on the participant enrollment form (ie, demographics, food behavior checklist, and detailed 24-hour dietary recall with sufficient detail). In addition, staff are required to use a progression record to document lesson plans, narrative notes regarding participant behavior change in each session, and attendance. The completeness of the required data elements, along with the detail provided in narrative notes and amendments made to lesson plans, provides clues to the quality of programming and reliability of recorded data. Incomplete or questionable records provide a starting point for discussion and indications of where training is needed.
Final report
Notes and impressions are compared and synthesized by the team in preparation for developing the final report. The written report provides documentation of these observations and findings. It emphasizes key findings, strengths, weaknesses, opportunities for improvement, and areas for further training/development. An important use for the report is that it provides background for assessing progress at future site visits. The report template form facilitates completion of the written summary of site visit findings and recommendations. Items in the report include the following:
- •“Executive Summary” highlights exceptional findings in the areas of management, recruitment, retention, supervision, programming, data management, or other areas as indicated by the site visit.
- •“Record Review Summary” provides a synopsis of findings and statistics.
- •“Discussion with Staff” highlights priorities, concerns, and issues raised by staff.
- •“Observation Summary” documents impressions of the educational delivery observed, use of resources, and participant reactions.
- •“Follow-up” outlines steps for bringing program policies and procedures into compliance with state guidelines and federal regulations, suggestions for in-service training, or additional visits needed to provide technical assistance aimed at improving program quality.
The report is drafted by the site visit team leader and shared with the county nutrition supervisor and campus and regional staff who participated in the visit. When comments from all parties have been received and the report edited, a final hard copy is sent to the nutrition supervisor, with copies to the SNAP-Ed manager in the NYS Office of Temporary and Disability Assistance, Cornell Food and Nutrition Education in Communities leadership, and the county extension executive director.
Follow-up for quality improvement
County nutrition supervisors are responsible for ensuring quality and bringing programs into compliance when problems related to federal requirements or state guidance have surfaced. In some cases, this may require only a review and reinforcement of established but poorly executed policies. In other cases, confusion or misunderstandings regarding expectations of staff require skill building in-services. For example, in 2006 one such training session was developed on the use and completion of progression records used to document a group education county program. In 2009, a skills training was developed to address collection of high-quality 24-hour dietary recalls in a county in which data had been insufficient for useful dietary assessment. Both of these in-services were developed for specific counties to address issues uncovered in site visits and were then adapted for statewide use. In 2008, combined feedback from a number of counties with relatively new nutrition supervisors (in positions less than 1 year) resulted in the development of a quarterly checklist to assist supervisors in tracking key tasks during the year. This tool lists ongoing program requirements, eg, report due dates, with space for notes and completion dates. The checklist has now become part of the training for new nutrition supervisors.
Annual objectives for site visits
Each year, specific objectives for the site visit process are identified by the Food and Nutrition Education in Communities team, with input from regional coordinators. The 2005 objectives included a review of site visit procedures, and the process was amended to routinely include regional staff. In subsequent years, objectives have included using regional staff in at least 1 review outside their resident region; review of staff training logs to understand staff development experiences; review of nutrition education materials to ensure that locally available materials were up to date, relevant, and approved; and review of marketing and recruiting materials to gain ideas about best practices at the county level in this challenging area. These amendments are driven by program priorities, discussion with the SNAP-Ed manager in the NYS Office of Disability Assistance, and regional coordinators.
Lessons Learned
This process has proved to be a valuable learning opportunity for all involved. Many internal values of program evaluation have been realized.
6
Interactions With Staff
The value and power of face-to-face interaction with staff at all levels cannot be overstated. The local presence of campus leadership emphasizes the importance of the programs and highlights the concern for all aspects of program delivery. Communication among campus faculty, local program managers, and the county executive directors has improved. The presence of state staff in the field provides access that is not otherwise available, particularly to paraprofessional educators who receive immediate feedback and dialogue about their work. Paraprofessional educators do not typically travel to campus for training sessions because these are provided regionally in NYS. Given the size of the state and large number of county staff, travel costs and travel time prevent all but the rare opportunity for most paraprofessional staff to interact with campus-based leadership. Local staff have embraced the opportunity to meet with the site visit team.
Program Policies and Procedures
Since the initiation of the formal site visit process, clarity and consistency of program policies and procedures have improved. A solid understanding of program operations is reinforced through discussion of recruitment, enrollment, nutrition education processes, and behavior change with all levels of staff. Although no preparation is required on the part of county staff, except for adjustments in their schedule to be present for the site visit, staff have been conducting internal reviews of their data and records before these visits. This preparatory step has unearthed issues that were previously not brought to the attention of the local supervisor. Frequently, steps are taken to remedy inappropriate practices before the site visit. Local supervisors report using this “teachable moment” to make necessary adjustments in local procedures. One positive outcome of this preliminary review is the opportunity for the supervisor to clarify issues when paraprofessional educators disagree, are uncertain about what to do, or have developed unique recordkeeping strategies. In many instances, the supervisor requests that campus staff reinforce policies during the site visit.
The contribution of each staff member to the total state program is understood better as a result of this site visit process. Discussions of county-level data typically elicit questions about how state behavior change data are subsequently used for reporting to the federal level. These discussions and realization on the part of county frontline staff that their contribution to the behavior change achieved by their participants affects the national program elevate the importance of quality nutrition education, accurate recordkeeping, and continuous training.
Staff Training
The site visit process has led to more precisely targeted staff training. Observation of staff in the field has provided insight into how well staff have been prepared for their jobs. Much effort has gone into initial nutrition staff training in Cornell Cooperative Extension. New paraprofessional educators participate in 19 days of training, complete a variety of follow-up activities, and are required to meet with their supervisors during their first year in a process of supported transition into more independent activities.
7
Clarification of paraprofessional educators’ need for further training or mentoring is a significant aspect of the supervisor’s role in supported transition.7
Firsthand observation by state leadership allows an opportunity to assess the quality of participant/staff interactions and assess the value and effectiveness of staff preparation and training, providing input and support to local managers.Involvement of Regional Coordinators
Regional coordinators’ participation in site visits outside their region has broadened their view of program possibilities and facilitated transfer of new and innovative ideas across regional boundaries. Regional cohorts have developed into effective, supportive, and innovative working groups.
Implications for Research and Practice
Few studies have reported on factors related to quality service specific to extension nutrition programs, although Dickin et al
8
described a positive effect of good management on participant outcomes in EFNEP. Research on a number of factors related to quality service will be important, including qualitative research by a third party on perception of local staff of the usefulness of the site visit process. Additional research is also needed to better define practices that are consistent with the highest-quality nutrition programs, eg, the optimum number of educational sessions to achieve the best cost-effectiveness, best practices in managing small sites with few educators, and regional management of paraprofessional educators.Improving program quality is an iterative process. From the state perspective, meeting program requirements is the minimum expectation of a county; high-quality programs will exceed this minimum. Regular, standardized onsite visits can ensure quality by increasing communication and trust between county and state staff, providing an understanding of the challenges educators face, and uncovering issues with best practices and noncompliance to policies, all of which are essential to ensure excellence across the state.
9
Excellence in nutrition programming embraces all that is desirable in customer service, competence and knowledge of the staff, and support and reinforcement from the state. The current climate of diminishing resources will require effective management practices at all levels to ensure and maintain excellence in programming.
References
- Evaluating food stamp nutrition education: a view from the field of program evaluation.J Nutr Educ Behav. 2006; 38: 12-17
US Department of Agriculture. Expanded Food and Nutrition Education Program Policies. http://www.csrees.usda.gov/nea/food/efnep/pdf/program-policy.pdf. Published October 1983. Accessed June 28, 2011.
Food and Nutrition Service, US Department of Agriculture. Food Stamp Nutrition Education Guiding Principles. 2009. http://www.fns.usda.gov/ora/menu/FSNE/GuidingPrinciples.pdf.Published, 2011 Published 2009. Accessed June 28, 2011.
- Challenges and strategies in applying performance measurement to federal public health programs.Eval Program Plann. 2010; 33: 365-372
- Food Stamp Nutrition Education Management Evaluation Review Guide For State Management Evaluation Reviewers.US Department of Agriculture, Food and Nutrition Service, Washington, DC2007
- Evaluating food stamp nutrition education: a view from the field of program evaluation.J Nutr Educ Behav. 2006; 38: 12-17
- Maximizing Paraprofessional Potential.Keieger Publishing Co, Malabar, FL1998
- Behavior change among EFNEP beneficiaries is higher in well-managed sites where front-line nutrition educators value the program.J Nutr. 2005; 135: 2199-2205
- A review of performance measurement: towards performance management.Computers Industry. 2005; 56: 663-680
Article info
Publication history
Published online: October 17, 2011
Identification
Copyright
© 2012 Society for Nutrition Education and Behavior. Published by Elsevier Inc. All rights reserved.
