An Evaluation of Inter-coder and Intra-coder Reliability for 24-Hour Dietary Recall Data Entered in WebNEERS

Published:February 06, 2019DOI:



      To evaluate inter-coder (between-coder) and intra-coder (within-coder) reliability among trained data coders who enter 24-hour dietary recall data collected through Expanded Food and Nutrition Education Program operations in the state of Georgia.


      This study employed multiple cross-sectional evaluations of inter-coder reliability and a short-term longitudinal evaluation of intra-coder reliability.


      Study participants consisted of trained data coders (n = 9) who were employed during the 12-month period of evaluation.

      Main Outcome Measures

      Primary outcome measures were inter-coder and intra-coder reliability across data entered into the Web-based Nutrition Education Evaluation and Reporting System. Statistical analyses were conducted using IBM SPSS 24. Descriptive statistics were generated and inter-coder and intra-coder reliability were assessed using 2-way mixed intraclass correlation coefficients.


      Results of this evaluation indicated good to excellent inter-coder reliability among all coders, and excellent intra-coder reliability among the majority of coders. However, some notable inconsistencies were identified within the intra-coder reliability analyses.


      Future strategies to improve data quality within Expanded Food and Nutrition Education Program operations might include enhanced training for data coders, implementation of error detection protocols, expansion of the Web-based Nutrition Education Evaluation and Reporting System database, and exploration of automated, computer-assisted administration of 24-hour dietary recalls.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access

      SNEB Member Login

      SNEB Members, full access to the journal is a member benefit. Login via the SNEB Website to access all journal content and features.


      Subscribe to Journal of Nutrition Education and Behavior
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Chipman H
        • Kendall PA
        20years of EFNEP: changes and challenges.
        J Nutr Educ Behav. 1990; 21: 265-269
        • Gills SM
        • Baker SS
        • Auld G
        Collection methods for the 24-hour dietary recall as used in the Expanded Food and Nutrition Education Program.
        J Nutr Educ Behav. 2017; 49 (250–256.e251)
      1. US Department of Agriculture, National Institute of Food and Agriculture. https://nifa.usda.ogv/tool/webneers. Accessed January 4, 2018.

        • Ahluwalia N
        • Dwyer J
        • Terry A
        • Moshfegh A
        • Johnson C
        Update on NHANES dietary data: focus on collection, release, analytical considerations, and uses to inform public policy.
        Adv Nutr. 2016; 7: 121-134
        • Townsend MS
        • Ganthavorn C
        • Neelon M
        • Donohue S
        • Johns MC
        Improving the quality of data from EFNEP participants with low literacy skills: a participant-driven model.
        J Nutr Educ Behav. 2014; 46: 309-314
        • Barchard KA
        • Pace LA
        Preventing human error: the impact of data entry methods on data accuracy and statistical results.
        Comput Hum Behav. 2011; 27: 1834-1839
      2. US Department of Health and Human Services, US Department of Agriculture. 2015–2020 Dietary Guidelines for Americans. 8th edition. December 2015. Accessed January 31, 2019

        • Cicchetti D
        Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.
        Psychol Assess. 1994; 6: 284-290
        • Dolecek TA
        • Stamler J
        • Caggiula AW
        • Tillotson JL
        • Buzzard IM
        Methods of dietary and nutritional assessment and intervention and other methods in the Multiple Risk Factor Intervention Trial.
        Am J Clin Nutr. 1997; 65: 196S-210S
        • Kohlmeier L
        • Mendez M
        • McDuffie J
        • Miller M
        Computer-assisted self-interviewing: a multimedia approach to dietary assessment.
        Am J Clin Nutr. 1997; 65: 1275s-1281s
        • Subar AF
        • Kirkpatrick SI
        • Mittl B
        • et al.
        The Automated Self-Administered 24-hour dietary recall (ASA24): a resource for researchers, clinicians, and educators from the National Cancer Institute.
        J Acad Nutr Diet. 2012; 112: 1134-1137
        • Thompson FE
        • Dixit-Joshi S
        • Potischman N
        • et al.
        Comparison of interviewer-administered and automated self-administered 24-hour dietary recalls in 3 diverse integrated health systems.
        Am J Epidemiol. 2015; 181: 970-978
        • Kirkpatrick SI
        • Subar AF
        • Douglass D
        • et al.
        Performance of the Automated Self-Administered 24-hour Recall relative to a measure of true intakes and to an interviewer-administered 24-h recall.
        Am J Clin Nutr. 2014; 100: 233-240
        • Willett W.
        Nutritional Epidemiology.
        2nd ed. Oxford University Press, New York, NY1998
        • Lee-Kwan SH
        • Moore LV
        • Blanck HM
        • Harris DM
        • Galuska D
        Disparities in state-specific adult fruit and vegetable consumption—United States, 2015.
        MMWR Morb Mortal Wkly Rep. 2017; 66: 1241-1247
        • McCullough ML
        • Karanja NM
        • Lin PH
        • et al.
        Comparison of 4 nutrient databases with chemical composition data from the Dietary Approaches to Stop Hypertension trial. DASH Collaborative Research Group.
        J Am Diet Assoc. 1999; 99: S45-S53
        • Gisev N
        • Bell JS
        • Chen TF
        Interrater agreement and interrater reliability: key concepts, approaches, and applications.
        Res Social Adm Pharm. 2013; 9: 330-338
        • Bartlett JW
        • Frost C.
        Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables.
        Ultrasound Obstet Gynecol. 2008; 31: 466-475
        • Needham DM
        • Sinopoli DJ
        • Dinglas VD
        • et al.
        Improving data quality control in quality improvement projects.
        Int J Qual Health Care. 2009; 21: 145-150