A Reviewer’s Guide for Appraising Questionnaire Methods Papers
INTRODUCTION
Define the topic and the research question that the questionnaire addresses
___ Clearly specifies the topic of interest, the target audience, and the primary and secondary objectives to be addressed
___ Context (and citations) that indicates what is known about the topic and the rationale is provided for the development of the survey and why a new instrument (or adaptation of an existing one) is necessary.
METHOD
Questionnaire/item development
___ A systematic approach was used to develop the questionnaire including citations for survey methodology and conduct. If an existing tool was used (in whole or in part), its previously reported psychometric properties should be described and references provided to the original development work. Which parts of the instrument to be utilized should be described.
___ Constructs to be studied are defined
___ Description of item development and how items were related to underlying theory
___ Description of how response options were selected
___ Consideration of subject burden in design, length, and administration of the survey
___ How questionnaires were formatted and design principles invoked
___ Pretesting of items; by and with whom, and the process used to pretest them (e.g., cognitive interviewing)
___ Scoring scheme adequately described such that reader could use the questionnaire
___ Questionnaire pilot-tested and psychometric properties (aspects of reliability and validity) evaluated and reported
Questionnaire testing—recruitment, sampling, bias
___ Target population specified and the sample representative of the population to be studied
___ Sampling approaches described (in person, web, multimodal); what sampling approaches were used? (Single? Multimodal [some web; and other])
___Sampling frame specified (how participants identified and recruited into the sample) and was oversampling in a specific group or access point described. Description of how sampling error (difference between sample and the target population) was minimized. Sampling described (random, probability, clustering, stratification, etc.)
___Description of whether weighting of sample (and on which characteristic) was adopted to achieve results from a representative sample
___ Justification of sample size including the process to estimate the necessary sample size, including characteristics, if any, upon which the sample was stratified
___ Description of questionnaire administration and whether administration was appropriate for the research objective or question posed (eg, aural vs. visual administration; web-based, pen and paper, read to respondents, languages)
___ Details regarding prenotification, use of a cover letter, number of attempts to contact, and an incentive for questionnaire completion
Data analysis
___ Response rate reported including strategies used to optimize the response rate; how was response rate define and the criterion for response rate (citation included)
___ Description of the statistical analysis conducted including software package utilized, item analysis conducted to detect nonvalid or patterned responses (mechanical respondents), missingness analysis, imputation of responses when utilized, questionnaire completion time
___ Statistical analyses or tests performed and the criteria associated with these tests
RESULTS
___ Results clearly and transparently reported
___ Results align with and address the questionnaire objectives
___ Demographic data of the questionnaire respondents provided and the response options for demographic items specified
___ Evidence for validity and reliability provided
___ Results succinctly summarized
___ Interpretations of the results align with the data presented. Significance vs. Importance presented
DISCUSSION
___ Findings placed within the context of the existing literature
___ Presents a compelling case that the instrument in question can be used in other samples and can be expected to perform similarly. That is, results are generalizable
___ Replication possible by other researchers given information provided
Provides comparison with literature related to similar questionnaire development papers
___ Limitations of the sample and sampling frame used, and the potential for response or nonresponse bias discussed.
IMPLICATIONS FOR RESEARCH AND PRACTICE
___ Recommendations for ongoing development of the instrument
___ Implications for use in either clinical or other research studies and populations
___ How instrument can inform policy development
___ Questionnaire provided in its entirety (as an electronic appendix or in print)
