Questionnaire Development Research Methods| Volume 54, ISSUE 4, P346-358, April 2022

Download started.


Development of Reliable and Valid Questions to Assess Food Resource Management Behaviors in Adults With Limited Income

Published:February 04, 2022DOI:



      Develop reliable, valid questions to assess changes in food resource management (FRM) behaviors in adults with limited incomes.


      Questionnaire development using a mixed-methods approach: content validity (subject matter and curricula), face validity, temporal reliability (test-retest), sensitivity to change, and exploratory factor analysis (EFA).


      Community settings in 12 states.


      Convenience samples of English-speaking Expanded Food and Nutrition Education Program (EFNEP) or EFNEP-eligible adults: 105 (cognitive interviews), 181 (test-retest), 185 (sensitivity), and 389 (EFA) adults.

      Variables Measured

      Behaviors related to FRM skills: planning, shopping, and budgeting.


      Consistency and agreement in cognitive interviews and temporal reliability; sensitivity at posttest (paired t tests, Wilcoxon signed-rank tests); internal consistency of scales identified in EFA (Cronbach α). P < 0.5.


      All questions had acceptable temporal reliability ranges for the intraclass correlation coefficient (0.48–0.74) and Spearman rank-order correlation (0.48–0.73). All questions were sensitive to change at posttest (P < 0.001). Planning and saving scales, revealed by EFA, demonstrated internal consistency (> 0.80 Cronbach α).

      Conclusions and Implications

      The 9 FRM behavior questions have acceptable temporal reliability and content and face validity and can be used nationally by EFNEP to assess participants’ self-reported behavior changes. Other nutrition programs with similar audiences and content could use these questions to measure changes in FRM behaviors.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access

      SNEB Member Login

      SNEB Members, full access to the journal is a member benefit. Login via the SNEB Website to access all journal content and features.


      Subscribe to Journal of Nutrition Education and Behavior
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Kumcu A, Kaufman P. Food spending adjustments during recessionary times. Accessed September 8, 2021.

        • Patrick SW
        • Henkhaus LE
        • Zickafoose JS
        • et al.
        Well-being of parents and children during the COVID-19 pandemic: a national survey.
        Pediatrics. 2020; 146e2020016824
        • Abrams SA
        • Avalos A
        • Gray M
        • Hawthorne KM.
        High level of food insecurity among families with children seeking routine care at federally qualified health centers during the coronavirus disease 2019 pandemic.
        J Pediatr X. 2020; 4100044
        • Coleman-Jensen A
        • Rabbitt MP
        • Gregory CA
        • Singh A.
        Household Food Security in the United States in 2019.
        US Department of Agriculture, Economic Research Service, 2020 (Economic Research Report Number 275.) (Accesed September 8, 2021)
      2. Fitzpatrick KM, Harris C, Drawve G. Assessing U.S. food insecurity in the United States during COVID-19 pandemic. University of Arkansas; 2020. Accessed August 22, 2021.

        • Gundersen CG
        • Garasky SB.
        Financial management skills are associated with food insecurity in a sample of households with children in the United States.
        J Nutr. 2012; 142: 1865-1870
        • O'Neill B
        • Xiao JJ
        • Ensle K.
        Improving health and increasing wealth: research insights and program outreach.
        J Natl Ext Assoc Fam Consumer Sci. 2017; 12: 11-23
        • Dollahite J
        • Olson C
        • Scott-Pierce M.
        The impact of nutrition education on food insecurity among low-income participants in EFNEP.
        Fam Consum Sci Res J. 2003; 32: 127-139
        • Adedokun OA
        • Plonski P
        • Aull M.
        Food resource management mediates the relationship between participation in a SNAP-Ed nutrition education program and diet quality.
        J Nutr Educ Behav. 2021; 53: 401-409
        • Hersey J
        • Anliker J
        • Miller C
        • et al.
        Food shopping practices are associated with dietary quality in low-income households.
        J Nutr Educ Behav. 2001; 33 (suppl 1): S16-S26
      3. US Department of Agriculture, National Institute of Food and Agriculture. Community Nutrition Education Logic Model (CNE). Accessed September 8, 2021.

        • US Department of Agriculture, National Institute of Food and Agriculture
        The Expanded Food and Nutrition Education Program Policies.
        National Institute of Food and Agriculture, 2017 (Accessed September 8, 2021)
        • US Department of Agriculture, US Department of Health and Human Services
        Dietary Guidelines for Americans, 2020–2025.
        9th ed. US Department of Agriculture, 2020 (Accessed September 8, 2021)
        • US Department of Health and Human Services
        Physical Activity Guidelines for Americans.
        2nd ed. US Department of Health and Human Services, 2018 (Accessed September 8, 2021)
      4. US Department of Health and Human Services. Keep food safe. Accessed September 8, 2021.

        • Hoerr SL
        • Abdulkadri AO
        • Miller SR
        • et al.
        Improving measurement of the EFNEP outcomes using factor analysis of the Behavior Checklist.
        J Ext. 2011; 49 (Article 6)
        • Murray EK
        • Auld G
        • Baker SS
        • et al.
        Methodology for developing a new EFNEP food and physical activity behaviors questionnaire.
        J Nutr Educ Behav. 2017; 49 (777–783.e1)
        • Murray EK
        • Baker SS
        • Betts NM
        • Hess A
        • Auld G.
        Development of a national dietary behaviors questionnaire for EFNEP adult participants.
        J Nutr Educ Behav. 2020; 52: 1088-1099
        • Moore CJ
        • Sweet CL
        • Harrison JA
        • Franck KL.
        Validating responses to a food safety survey with observations of food preparation behaviors among limited resource populations.
        Food Prot Trends. 2019; 39: 449-460
        • US Department of Agriculture, Research, Education, and Economics
        • National Institute of Food and Agriculture
        EFNEP Basics for New Coordinators: the Expanded Food and Nutrition Education Program (EFNEP).
        US Department of Agriculture, 2014 (Accessed September 8, 2021)
        • Darko J
        • Eggett DL
        • Richards R.
        Shopping behaviors of low-income families during a 1-month period of time.
        J Nutr Educ Behav. 2013; 45: 20-29
        • Dinkins JM.
        Food preparers: their food budgeting, cost-cutting, and meal planning practices.
        Fam Econ Nutr Rev. 1997; 10: 34-37
        • Murray EK
        • Auld G
        • Inglis-Widrick R
        • Baker S.
        Nutrition content in a national nutrition education program for low-income adults: content analysis and comparison with the 2010 Dietary Guidelines for Americans.
        J Nutr Educ Behav. 2015; 47 (566–573.e1)
      5. Expanded Food and Nutrition Education Program (EFNEP) digital resources. Adult evaluation archive. US Department of Agriculture; 2012. Accessed September 8, 2021.

      6. Wardlaw MK, Hanula G. EFNEP Behavior Checklist Review. US Department of Agriculture; 2012. Accessed September 9, 2021.

        • Baranowski T
        • Missaghian M
        • Broadfoot A
        • et al.
        Fruit and vegetable shopping practices and social support scales: a validation.
        J Nutr Educ Behav. 2006; 38: 340-351
        • Hartline-Grafton H
        • Nyman R
        • Briefel R
        • et al.
        Prototype Notebook: Short Questions on Dietary Intake, Knowledge, Attitudes and Behaviors.
        US Department of Agriculture, Economic Research Service, Food and Rural Economics Division, 2004 (Accessed September 8, 2021)
        • McClelland JW
        • Keenan DP
        • Lewis J
        • et al.
        Review of evaluation tools used to assess the impact of nutrition education on dietary intake and quality, weight management practices, and physical activity of low-income audiences.
        J Nutr Educ. 2001; 33: S35-S48
        • Cabili C
        • Cohen R
        • Briefel R
        • Grau E.
        Development of a Questionnaire to Assess the Dietary Behavior of Low-Income Populations.
        Economic Research Service, US Department of Agriculture, 2009 (Accessed September 8, 2021)
        • Boateng GO
        • Neilands TB
        • Frongillo EA
        • Melgar-Quiñonez HR
        • SL Young
        Best practices for developing and validating scales for health, social, and behavioral research: a primer.
        Front Public Health. 2018; 6: 149
        • DeVellis R.
        Scale Development: Theory and Applications.
        3rd ed. SAGE Publications, Inc, 2011
        • Townsend MS.
        Evaluating Food Stamp Nutrition Education: process for development and validation of evaluation measures.
        J Nutr Educ Behav. 2006; 38: 18-24
        • Glasgow RE
        • Ory MG
        • Klesges LM
        • Cifuentes M
        • Fernald DH
        • Green LA.
        Practical and relevant self-report measures of patient health behaviors for primary care research.
        Ann Fam Med. 2005; 3: 73-81
        • Townsend MS
        • Kaiser LL.
        Brief psychosocial fruit and vegetable tool is sensitive for the US Department of Agriculture's nutrition education programs.
        J Am Diet Assoc. 2007; 107: 2120-2124
        • Beatty PC
        • Willis GB.
        Research Synthesis: the practice of cognitive interviewing.
        Public Opin Q. 2007; 71: 287-311
      7. Willis G. Cognitive Interviewing and Questionnaire Design: A Training Manual. US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. 1994. Working Paper Series, No. 7. American Statistical Association; 1999. Accessed September 9, 2021.

        • Banna JC
        • Townsend MS.
        Assessing factorial and convergent validity and reliability of a food behaviour checklist for Spanish-speaking participants in US Department of Agriculture nutrition education programmes.
        Public Health Nutr. 2011; 14: 1165-1176
        • Widaman KF.
        Exploratory factor analysis and confirmatory factor analysis.
        in: Cooper H Camic PM Long DL Panter AT Rindskopf D Sher KJ APA Handbook of Research Methods in Psychology: Data Analysis and Research Publication Vol. 3. American Psychological Association, 2012: 361-390
        • Rouquette A
        • Falissard B.
        Sample size requirements for the internal validation of psychiatric scales.
        Int J Methods Psychiatr Res. 2011; 20: 235-249
        • Comrey AL.
        A First Course in Factor Analysis.
        Academic Press, 1973
        • Streiner DL
        • Norman GR
        • Cairney J.
        Health Measurement Scales: A Practical Guide to Their Development and Use.
        5th ed. Oxford University Press, 2015
        • Yen M
        • Lo LH.
        Examining test–retest reliability: an intra-class correlation approach.
        Nurs Res. 2002; 51: 59-62
        • Guadagnoli E
        • Velicer WF.
        Relation of sample size to the stability of component patterns.
        Psychol Bull. 1988; 103: 265-275
        • Kaiser HF.
        The application of electronic computers to factor analysis.
        Educ Psychol Meas. 1960; 20: 141-151
        • Sim J
        • Wright C.
        Research in Health Care: Concepts, Designs and Methods.
        Stanley Thornes, 2000
        • Chan LL
        • Idris N.
        Validity and reliability of the instrument using exploratory factor analysis and Cronbach alpha.
        Int J Acad Res Bus Soc Sci. 2017; 7: 400-410
        • Schlomer GL
        • Bauman S
        • Card NA.
        Best practices for missing data management in counseling psychology.
        J Couns Psychol. 2010; 57: 1-10
        • Shrout PE.
        Measurement reliability and agreement in psychiatry.
        Stat Methods Med Res. 1998; 7: 301-317
        • Campbell CC
        • Desjardins E.
        A model and research approach for studying the management of limited food resources by low income families.
        J Nutr Educ. 1989; 21: 162-171
        • Lavelle F
        • McGowan L
        • Hollywood L
        • et al.
        The development and validation of measures to assess cooking skills and food skills.
        Int J Behav Nutr Phys Act. 2017; 14: 118
        • Dubowitz T
        • Cohen DA
        • Huang CY
        • Beckman RA
        • Collins RL.
        Using a grocery list is associated with a healthier diet and lower BMI among very high-risk adults.
        J Nutr Educ Behav. 2015; 47: 259-264
        • Ducrot P
        • Méjean C
        • Aroumougame V
        Meal planning is associated with food variety, diet quality and body weight status in a large sample of French adults.
        Int J Behav Nutr Phys Act. 2017; 14: 12
        • Townsend MS
        • Kaiser LL
        • Allen LH
        • Joy AB
        • Murphy SP.
        Selecting items for a food behavior checklist for a limited-resource audience.
        J Nutr Educ Behav. 2003; 35: 69-77
        • Dickin KL
        • Lent M
        • Lu AH
        • Sequeira J
        • Dollahite JS.
        Developing a measure of behavior change in a program to help low-income parents prevent unhealthful weight gain in children.
        J Nutr Educ Behav. 2012; 44: 12-21
        • Zemeir LA
        • Butler J
        • Howard DE.
        Validity and reliability of the Healthy Families Survey: a key component of the Maryland Supplemental Nutrition Assistance Program–Education (SNAP-Ed) evaluation.
        J Nutr Educ Behav. 2018; 50: 632-637
        • Banna JC
        • Buchthal OV
        • Tauyan S.
        Assessing face validity of a food behavior checklist for limited-resource Filipinos.
        Hawaii J Med Public Health. 2015; 74: 334-340
        • Kennedy LG
        • Kichler EJ
        • Seabrook JA
        • Matthews JI
        • Dworatzek PDN.
        Validity and reliability of a food skills questionnaire.
        J Nutr Educ Behav. 2019; 51: 857-864
        • Barton KL
        • Wrieden WL
        • Anderson AS.
        Validity and reliability of a short questionnaire for assessing the impact of cooking skills interventions.
        J Hum Nutr Diet. 2011; 24: 588-595
        • Green CM
        • Berkule SB
        • Dreyer BP
        • et al.
        Maternal literacy and associations between education and the cognitive home environment in low-income families.
        Arch Pediatr Adolesc Med. 2009; 163: 832-837
        • Townsend MS
        • Ganthavorn C
        • Neelon M
        • Donohue S
        • Johns MC.
        Improving the quality of data from EFNEP participants with low literacy skills: a participant-driven model.
        J Nutr Educ Behav. 2014; 46: 309-314
        • Sylvia LG
        • Bernstein EE
        • Hubbard JL
        • Keating L
        • Anderson EJ.
        Practical guide to measuring physical activity.
        J Acad Nutr Diet. 2014; 114: 199-208
        • Rohs FR
        • Langone CA
        • Coleman RK.
        Response shift bias: a problem in evaluating nutrition training using self-report measures.
        J Nutr Educ. 2001; 33: 165-170
        • Miller TM
        • Abdel-Maksoud MF
        • Crane LA
        • Marcus AC
        • Byers TE.
        Effects of social approval bias on self-reported fruit and vegetable consumption: a randomized controlled trial.
        Nutr J. 2008; 7: 18
      8. Expanded Food and Nutrition Education Program (EFNEP). Adult Questionnaire. US Department of Agriculture, National Institute of Food and Agriculture; 2020. Accessed September 8, 2021.