A Reviewer’s Guide for Appraising Questionnaire Methods Papers


Define the topic and the research question that the questionnaire addresses

___ Clearly specifies the topic of interest, the target audience, and the primary and secondary objectives to be addressed

___ Context (and citations) that indicates what is known about the topic and the rationale is provided for the development of the survey and why a new instrument (or adaptation of an existing one) is necessary.


Questionnaire/item development

___ A systematic approach was used to develop the questionnaire including citations for survey methodology and conduct.  If an existing tool was used (in whole or in part), its previously reported psychometric properties should be described and references provided to the original development work.  Which parts of the instrument to be utilized should be described.

___ Constructs to be studied are defined

___ Description of item development and how items were related to underlying theory

___ Description of how response options were selected

___ Consideration of subject burden in design, length, and administration of the survey

___ How questionnaires were formatted and design principles invoked

___ Pretesting of items; by and with whom, and the process used to pretest them (e.g., cognitive interviewing)

___ Scoring scheme adequately described such that reader could use the questionnaire

___ Questionnaire pilot-tested and psychometric properties (aspects of reliability and validity) evaluated and reported

Questionnaire testing—recruitment, sampling, bias

___ Target population specified and the sample representative of the population to be studied

___ Sampling approaches described (in person, web, multimodal); what sampling approaches were used? (Single? Multimodal [some web; and other])

___Sampling frame specified (how participants identified and recruited into the sample) and was oversampling in a specific group or access point described. Description of how sampling error (difference between sample and the target population) was minimized. Sampling described (random, probability, clustering, stratification, etc.)

___Description of whether weighting of sample (and on which characteristic) was adopted to achieve results from a representative sample

___ Justification of sample size including the process to estimate the necessary sample size, including characteristics, if any, upon which the sample was stratified

___ Description of questionnaire administration and whether administration was appropriate for the research objective or question posed (eg, aural vs. visual administration; web-based, pen and paper, read to respondents, languages)

___ Details regarding prenotification, use of a cover letter, number of attempts to contact, and an incentive for questionnaire completion

Data analysis

___ Response rate reported including strategies used to optimize the response rate; how was response rate define and the criterion for response rate (citation included)

___ Description of the statistical analysis conducted including software package utilized, item analysis conducted to detect nonvalid or patterned responses (mechanical respondents), missingness analysis, imputation of responses when utilized, questionnaire completion time

___ Statistical analyses or tests performed and the criteria associated with these tests


___ Results clearly and transparently reported

___ Results align with and address the questionnaire objectives

___ Demographic data of the questionnaire respondents provided and the response options for demographic items specified

___ Evidence for validity and reliability provided

___ Results succinctly summarized

___ Interpretations of the results align with the data presented. Significance vs. Importance presented


___ Findings placed within the context of the existing literature

___ Presents a compelling case that the instrument in question can be used in other samples and can be expected to perform similarly.  That is, results are generalizable

___ Replication possible by other researchers given information provided

      Provides comparison with literature related to similar questionnaire development papers

___ Limitations of the sample and sampling frame used, and the potential for response or nonresponse bias discussed.


___ Recommendations for ongoing development of the instrument

___ Implications for use in either clinical or other research studies and populations

___ How instrument can inform policy development

___ Questionnaire provided in its entirety (as an electronic appendix or in print)