Research Article| Volume 48, ISSUE 7, P437-452.e1, July 2016

Validity and Reliability of Behavior and Theory-Based Psychosocial Determinants Measures, Using Audience Response System Technology in Urban Upper-Elementary Schoolchildren

Published:April 30, 2016DOI:



      To examine the validity and reliability of a questionnaire administered with an audience response system (ARS).


      Cross-sectional study.


      Two New York City public elementary schools.


      Fourth- and fifth-grade students.

      Main Outcome Measures

      A Food, Health, and Choices questionnaire (FHC-Q) assessed energy balance-related behaviors (EBRBs) including intake of fruits and vegetables, sugar-sweetened beverages, processed packaged snacks, and fast food; physical activity; recreational screen time; and associated psychosocial determinants (≥ 3 questions/outcome scale). Previously validated reference instruments were used for relative validation. The ARS format was compared with a paper-and-pencil format. All measures were administered in a classroom setting.


      Pearson correlation coefficients between the reference instruments and the FHC-Q were calculated. Internal consistency reliabilities were evaluated with Cronbach α. Spearman rank correlation, intra-class correlation, and percent agreement were used for test-retest reliability between paper-and-pencil and ARS, and between 2 ARS FHC-Q administrations.


      Correlations for EBRBs with reference instruments ranged from 0.38 to 0.61 (P < .01). Cronbach α ranged from .77 to .92 for EBRBs and .74 to .90 for psychosocial determinants. Test-retest reliability correlations ranged from 0.36 to 0.87 (P < .001). Agreement for knowledge questions ranged from 69.8% to 84.8%.

      Conclusions and Implications

      The ARS FHC-Q has acceptable validity and reliability for collecting data on EBRBs and associated psychosocial determinants for upper-elementary students.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access

      SNEB Member Login

      SNEB Members, full access to the journal is a member benefit. Login via the SNEB Website to access all journal content and features.


      Subscribe to Journal of Nutrition Education and Behavior
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Diet, nutrition and the prevention of chronic diseases.
        World Health Organ Tech Rep Ser. 2003; 916 (1–149, back cover): i-viii
        • Summerbell C.D.
        • Douthwaite W.
        • Whittaker V.
        • et al.
        The association between diet and physical activity and subsequent excess weight gain and obesity assessed at 5 years of age or older: a systematic review of the epidemiological evidence.
        Int J Obes (Lond). 2009; 33: S1-S92
        • Barlow S.E.
        • Expert C.
        Expert committee recommendations regarding the prevention, assessment, and treatment of child and adolescent overweight and obesity: summary report.
        Pediatrics. 2007; 120: S164-S192
        • Khambalia A.Z.
        • Dickinson S.
        • Hardy L.L.
        • Gill T.
        • Baur L.A.
        A synthesis of existing systematic reviews and meta-analyses of school-based behavioural interventions for controlling and preventing obesity.
        Obes Rev. 2012; 13: 214-233
        • Contento I.R.
        • Koch P.A.
        • Lee H.
        • Calabrese-Barton A.
        Adolescents demonstrate improvement in obesity risk behaviors after completion of choice, control & change, a curriculum addressing personal agency and autonomous motivation.
        J Am Diet Assoc. 2010; 110: 1830-1839
        • Johnson B.T.
        • Scott-Sheldon L.A.
        • Carey M.P.
        Meta-synthesis of health behavior change meta-analyses.
        Am J Public Health. 2010; 100: 2193-2198
        • Wang Y.
        • Cai L.
        • Wu Y.
        • et al.
        What childhood obesity prevention programmes work? A systematic review and meta-analysis.
        Obes Rev. 2015; 16: 547-565
        • Baranowski T.
        • Anderson C.
        • Carmack C.
        Mediating variable framework in physical activity interventions: how are we doing? How might we do better?.
        Am J Prev Med. 1998; 15: 266-297
        • Baranowski T.
        • Lin L.S.
        • Wetter D.W.
        • Resnicow K.
        • Hearn M.D.
        Theory as mediating variables: why aren’t community interventions working as desired?.
        Annal Epidemiol. 1997; 7: S89-S95
        • Cerin E.
        • Barnett A.
        • Baranowski T.
        Testing theories of dietary behavior change in youth using the mediating variable model with intervention programs.
        J Nutr Educ Behav. 2009; 41: 309-318
        • Ajzen I.
        The theory of planned behavior.
        Org Behav Hum Decis. 1991; 50: 179-211
        • Bandura A.
        Health promotion by social cognitive means.
        Health Educ Behav. 2004; 31: 143-164
        • Park D.Y.
        Utilizing the Health Belief Model to predicting female middle school students’ behavioral intention of weight reduction by weight status.
        Nutr Res Pract. 2011; 5: 337-348
        • Ar-Yuwat S.
        • Clark M.J.
        • Hunter A.
        • James K.S.
        Determinants of physical activity in primary school students using the health belief model.
        J Multidiscip Healthc. 2013; 6: 119-126
        • Deci E.L.
        • Ryan R.M.
        The “what” and “why” of goal pursuits: human needs and the self-determination of behavior.
        Psychol Inquiry. 2000; 11: 227-268
        • Silva M.N.
        • Vieira P.N.
        • Coutinho S.R.
        • et al.
        Using self-determination theory to promote physical activity and weight control: a randomized controlled trial in women.
        J Behav Med. 2010; 33: 110-122
        • Baranowski T.
        • Klesges L.M.
        • Cullen K.W.
        • Himes J.H.
        Measurement of outcomes, mediators, and moderators in behavioral obesity prevention research.
        Prev Med. 2004; 38: S1-S13
        • Baranowski T.
        • Cullen K.W.
        • Baranowski J.
        Psychosocial correlates of dietary intake: advancing dietary intervention.
        Annu Rev Nutr. 1999; 19: 17-40
        • Di Noia J.
        • Contento I.R.
        Use of a brief food frequency questionnaire for estimating daily number of servings of fruits and vegetables in a minority adolescent population.
        J Am Diet Assoc. 2009; 109: 1785-1789
        • Singh A.S.
        • Vik F.N.
        • Chinapaw M.J.
        • et al.
        Test-retest reliability and construct validity of the ENERGY-child questionnaire on energy balance-related behaviours and their potential determinants: the ENERGY-project.
        Int J Behav Nutr Phys Act. 2011; 8: 136
        • Hedrick V.E.
        • Comber D.L.
        • Estabrooks P.A.
        • Savla J.
        • Davy B.M.
        The beverage intake questionnaire: determining initial validity and reliability.
        J Am Diet Assoc. 2010; 110: 1227-1232
        • Magarey A.
        • Watson J.
        • Golley R.K.
        • et al.
        Assessing dietary intake in children and adolescents: considerations and recommendations for obesity research.
        Int J Pediatr Obes. 2011; 6: 2-11
        • Thompson F.
        • Subar A.
        Dietary assessment methodology.
        in: Coulston A. Boushey C. Nutrition in the Prevention and Treatment of Disease. Academic Press, San Diego, CA2008: 5-9
        • McPherson R.S.
        • Hoelscher D.M.
        • Alexander M.
        • Scanlon K.S.
        • Serdula M.K.
        Dietary assessment methods among school-aged children: validity and reliability.
        Prev Med. 2000; 31: S11-S33
        • Neuhouser M.L.
        • Lilley S.
        • Lund A.
        • Johnson D.B.
        Development and validation of a beverage and snack questionnaire for use in evaluation of school nutrition policies.
        J Am Diet Assoc. 2009; 109: 1587-1592
        • Neuhouser M.L.
        • Patterson R.E.
        • Kristal A.R.
        • Eldridge A.L.
        • Vizenor N.C.
        A brief dietary assessment instrument for assessing target foods, nutrients and eating patterns.
        Public Health Nutr. 2001; 4: 73-78
        • Thiagarajah K.
        • Fly A.D.
        • Hoelscher D.M.
        • et al.
        Validating the food behavior questions from the elementary school SPAN questionnaire.
        J Nutr Educ Behav. 2008; 40: 305-310
        • Wilson A.M.
        • Magarey A.M.
        • Mastersson N.
        Reliability and relative validity of a child nutrition questionnaire to simultaneously assess dietary patterns associated with positive energy balance and food behaviours, attitudes, knowledge and environments associated with healthy eating.
        Int J Behav Nutr Phys Act. 2008; 5: 5
        • Donnelly R.
        • Springer A.
        Parental social support, ethnicity, and energy balance-related behaviors in ethnically diverse, low-income, urban elementary schoolchildren.
        J Nutr Educ Behav. 2015; 47: 10-18
        • Edmunds L.D.
        • Ziebland S.
        Development and validation of the Day in the Life Questionnaire (DILQ) as a measure of fruit and vegetable questionnaire for 7-9 year olds.
        Health Educ Res. 2002; 17: 211-220
        • De Bourdeaudhuij I.
        • Klepp K.I.
        • Due P.
        • et al.
        Reliability and validity of a questionnaire to measure personal, social and environmental correlates of fruit and vegetable intake in 10-11-year-old children in five European countries.
        Public Health Nutr. 2005; 8: 189-200
        • Hoelscher D.M.
        • Day R.S.
        • Kelder S.H.
        • Ward J.L.
        Reproducibility and validity of the secondary level School-Based Nutrition Monitoring student questionnaire.
        J Am Diet Assoc. 2003; 103: 186-194
        • Springer A.E.
        • Kelder S.H.
        • Byrd-Williams C.E.
        • et al.
        Promoting energy-balance behaviors among ethnically diverse adolescents: overview and baseline findings of The Central Texas CATCH Middle School Project.
        Health Educ Behav. 2013; 40: 559-570
        • Janssen E.H.
        • Singh A.S.
        • van Nassau F.
        • Brug J.
        • van Mechelen W.
        • Chinapaw M.J.
        Test-retest reliability and construct validity of the DOiT (Dutch Obesity Intervention in Teenagers) questionnaire: measuring energy balance-related behaviours in Dutch adolescents.
        Public Health Nutr. 2012; 17: 277-286
        • van Stralen M.M.
        • te Velde S.J.
        • Singh A.S.
        • et al.
        EuropeaN Energy balance Research to prevent excessive weight Gain among Youth (ENERGY) project: design and methodology of the ENERGY cross-sectional survey.
        BMC Public Health. 2011; 11: 65
        • Riebl S.K.
        • Paone A.C.
        • Hedrick V.E.
        • Zoellner J.M.
        • Estabrooks P.A.
        • Davy B.M.
        The comparative validity of interactive multimedia questionnaires to paper-administered questionnaires for beverage intake and physical activity: pilot study.
        JMIR Research Protoc. 2013; 2: e40
        • DeSorbo A.L.
        • Noble J.M.
        • Shaffer M.
        • Gerin W.
        • Williams O.A.
        The use of an audience response system in an elementary school-based health education program.
        Health Educ Behav. 2013; 40: 531-535
        • Caldwell J.E.
        Clickers in the large classroom: current research and best-practice tips.
        CBE- Life Sci Educ. 2007; 6: 9-20
        • Graham C.R.
        • Tripp T.R.
        • Seawright L.
        • Joeckel G.L.
        Empowering or compelling reluctant participators using audience response systems.
        Active Learning Higher Educ. 2007; 8: 233-258
        • Gamito E.J.
        • Burhansstipanov L.
        • Krebs L.U.
        • Bemis L.
        • Bradley A.
        The use of an electronic audience response system for data collection.
        J Cancer Educ. 2005; 20: 80-86
        • Blum J.W.
        • Jacobsen D.J.
        • Donnelly J.E.
        Beverage consumption patterns in elementary school aged children across a two-year period.
        J Am Coll Nutr. 2005; 24: 93-98
        • Brug J.
        • te Velde S.J.
        • Chinapaw M.J.
        • et al.
        Evidence-based development of school-based and family-involved prevention of overweight across Europe: the ENERGY-project’s design and conceptual framework.
        BMC Public Health. 2010; 10: 276
        • Kremers S.P.
        Theory and practice in the study of influences on energy balance-related behaviors.
        Patient Educ Counsel. 2010; 79: 291-298
        • McClain A.D.
        • Chappuis C.
        • Nguyen-Rodriguez S.T.
        • Yaroch A.L.
        • Spruijt-Metz D.
        Psychosocial correlates of eating behavior in children and adolescents: a review.
        Int J Behav Nutr Phys Act. 2009; 6: 54
        • Piernas C.
        • Popkin B.M.
        Trends in snacking among U.S. children.
        Health Aff (Millwood). 2010; 29: 398-404
        • Brug J.
        • de Vet E.
        • de Nooijer J.
        • Verplanken B.
        Predicting fruit consumption: cognitions, intention, and habits.
        J Nutr Educ Behav. 2006; 38: 73-81
        • Herrick K.A.
        • Rossen L.M.
        • Nielsen S.J.
        • Branum A.M.
        • Ogden C.L.
        Fruit consumption by youth in the United States.
        Pediatrics. 2015; 136: 664-671
        • Ervin R.B.
        • Kit B.K.
        • Carroll M.D.
        • Ogden C.L.
        Consumption of added sugar among U.S. children and adolescents, 2005-2008.
        NCHS Data Brief. 2012; 87: 1-8
        • Nielsen S.J.
        • Rossen L.M.
        • Harris D.M.
        • Odgen C.L.
        Fruit and vegetable consumption of U.S. Youth, 2009-2010.
        NCHS Data Brief. 2014; 156: 1-8
        • Majumdar D.
        • Koch P.A.
        • Lee H.
        • Contento I.R.
        • Islas-Ramos A.D.
        • Fu D.
        “Creature-101”: a serious game to promote energy balance-related behaviors among middle school adolescents.
        Games Health J. 2013; 2: 280-290
        • Bandura A.
        Social cognitive theory: an agentic perspective.
        Annu Rev Psychol. 2001; 52: 1-26
        • van Stralen M.M.
        • Yildirim M.
        • te Velde S.J.
        • et al.
        What works in school-based energy balance behaviour interventions and what does not? A systematic review of mediating mechanisms.
        Int J Obes (Lond). 2011; 35: 1251-1265
        • van der Horst K.
        • Oenema A.
        • van de Looij-Jansen P.
        • Brug J.
        The ENDORSE study: research into environmental determinants of obesity related behaviors in Rotterdam schoolchildren.
        BMC Public Health. 2008; 8: 142
        • Baxter S.D.
        • Smith A.F.
        • Litaker M.S.
        • Baglio M.L.
        • Guinn C.H.
        • Shaffer N.M.
        Children’s social desirability and dietary reports.
        J Nutr Educ Behav. 2004; 36: 84-89
        • Moore J.B.
        • Hanes Jr., J.C.
        • Barbeau P.
        • Gutin B.
        • Trevino R.P.
        • Yin Z.
        Validation of the Physical Activity Questionnaire for Older Children in children of different races.
        Pediatr Exerc Sci. 2007; 19: 6-19
        • Merriam S.B.
        Qualitative Research in Practice.
        Jossey-Bass, San Francisco, CA2002
        • Hu L.T.
        • Bentler P.M.
        Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
        Structural Equation Modeling. 1999; 6: 1-55
        • McGraw K.O.
        • Wong S.P.
        Forming inferences about some intraclass correlation coefficients.
        Psychological Methods. 1996; 1: 30-46
        • Streppel M.T.
        • de Vries J.H.
        • Meijboom S.
        • et al.
        Relative validity of the food frequency questionnaire used to assess dietary intake in the Leiden Longevity Study.
        Nutr J. 2013; 12: 75
        • Lance C.E.
        • Butts M.M.
        • Michels L.C.
        The sources of four commonly reported cutoff criteria: what did they really say?.
        Organizational Research Methods. 2006; 9: 202-220

      Linked Article