Discussion

The main objective of this review is to provide detailed data on the validation of SFFQ. A total of sixty SFFQ that met the selection criteria were found from several geographical regions. Europe is the region with the highest number of published studies (n 25)^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Klipstein-Grobusch, Den Breeijen and Goldbohm14–Reference Grootenhuis, Westenbrink and Sie36)} , while the number of studies from Latin America^{(Reference Dehghan, del Cerro and Zhang37,Reference Rodríguez, Méndez and Torún40,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45,Reference Hernández-Avila, Romieu and Parra47,Reference Macedo-Ojeda, Vizmanos-Lamotte and Márquez-Sandoval52)} has increased significantly in recent years. These data may indicate that the number of epidemiological assessments of diet in Latin America has probably increased recently since food-related health problems have become more prevalent in that region.

More than half of the studies (81·1 %) included male and female subjects. Some authors had specific reasons for selecting subjects of only one sex. For example, Willett et al. ^{(Reference Willet, Sampson and Stampfer42)} selected women because their study targeted female nurses; in the case of SFFQ administered only to men^{(Reference Mullie, Clarys and Hulens18,Reference Rimm, Giovannucci and Stampfer39,Reference Feskanich, Marshall and Rimm46)} , no reasons were given for this exclusive selection, with the exception of the study on male health professionals^{(Reference Rimm, Giovannucci and Stampfer39)}.

Regarding age, the broadest age ranges were reported in the studies by Gazan et al. ^{(Reference Gazan, Vieux and Darmon29)} and Shatenstein et al. ^{(Reference Shatenstein, Nadon and Godin41)}. The latter had the broadest range, with 64 years of difference between the lowest and highest values. The existence of such a broad age range in the SFFQ validation process supports the administration of this questionnaire to older adults and younger populations. Notice that it was decided to include articles with minimum ages below 18 years because the mean age was not influenced by these values^{(Reference Ramón, Micaló and Benítez22,Reference Chen, Ahsan and Parvez54,Reference Beck, Houston and McNaughton68)} .

Significant correlations were found regarding the analyses carried out for the validation of the SFFQ in the reviewed papers. Most of the studies used Pearson’s coefficients (68 %)^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12–Reference Knudsen, Hatch and Cueto15,Reference Moreira, Sampaio and Almeida17–Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Ramón, Micaló and Benítez22–Reference Rothenberg25,Reference Tjønneland, Overvad and Haraldsdóttir27,Reference Van Dongen, Lentjes and Wijckmans28,Reference Bonifacj, Gerber and Scali30,Reference Dumartheray, Krieg and Cornuz31,Reference Friis, Kjaer and Stripp35–Reference Rodríguez, Méndez and Torún40,Reference Willet, Sampson and Stampfer42,Reference Sanjeevi, Freeland-Graves and George44–Reference Longnecker, Lissner and Holden51,Reference Chen, Ahsan and Parvez54–Reference Jayawardena, Byrne and Soares56,Reference Sudha, Radhika and Sathya59,Reference Yang, Kim and Hwang61,Reference Whitton, Ho and Tay62,Reference Fayet, Flood and Petocz65,Reference Horwath66,Reference Gunes, Imeryuz and Akalin70)} . However, no consensus has been found in the literature concerning which statistical method is most suitable for assessing the validity of dietary tools^{(Reference Masson, MCNeill and Tomany16)}.

The Pearson’s (r) and Spearman’s (ρ) correlation coefficients measure the degree of linear association between two variables, the former being more suitable for normal distribution and the latter for non-normal distribution. However, neither of these coefficients provides information about the degree of agreement observed, nor about the presence of systematic differences between measurements or instruments^{(Reference Martínez, Tuya and Martínez71–Reference Martínez-González, Toledo, Sánchez-Villegas, Martínez-González, Sánchez-Villegas and Toledo-Atucha74)}. The intraclass correlation coefficient is usually used for this purpose in reproducibility studies^{(Reference Cortés-Reyes, Rubio-Romero and Gaitán-Duarte72,Reference Martínez-González, Toledo, Sánchez-Villegas, Martínez-González, Sánchez-Villegas and Toledo-Atucha74,Reference Pita and Pértegas75)} . However, it has been proposed as an alternative method to evaluate the agreement observed between methods in validation studies^{(Reference Fernández-Ballart, Piñol and Zazpe32,Reference Macedo-Ojeda, Vizmanos-Lamotte and Márquez-Sandoval52,Reference Cade, Thompson and Burley76)} . Taking these observations into consideration, it is suggested that validation studies include, in addition to the mean values of each method, the Pearson’s or Spearman’s correlation coefficients and, if desired, the intraclass correlation coefficient.

Regarding the interpretation of the correlation coefficients, Landis and Koch’s^{(Reference Jayawardena, Byrne and Soares56)} classification for the Kappa (κ) coefficient has been often used for this purpose. This classification is broken down as follows: 0 = poor agreement; 0·01–0·20 = slight agreement; 0·21–0·40 = fair agreement; 0·41–0·60 = moderate agreement; 0·61–0·80 = substantial agreement; 0·81–1·00 = almost perfect agreement. Other authors have suggested the following classification: 0 = none existent; r < 0·3 = poor; 0·30–0·70 = moderate; r > 0·70 = strong^{(Reference Sánchez-Villegas, Martínez-González, Martínez-González, Sánchez-Villegas and Toledo-Atucha77)}. The correlation coefficients in the reviewed studies suggest that validation may be possible, although the results of the same could vary from poor to excellent.

It should be noted that analyses adjusted by energy, sex or age may be added to the crude correlation analyses. The reason for the inclusion of these analyses is that nutritional consumption may be higher or lower depending on the dependent variable (e.g. energy). Most of the included studies adjust their analysis in accordance with energy consumption using the residual method^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Knudsen, Hatch and Cueto15–Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Ramón, Micaló and Benítez22,Reference Roddam, Spencer and Banks23,Reference Tjønneland, Overvad and Haraldsdóttir27,Reference Van Dongen, Lentjes and Wijckmans28,Reference Bonifacj, Gerber and Scali30–Reference Fidanza, Gentile and Porrini33,Reference Friis, Kjaer and Stripp35,Reference Dehghan, del Cerro and Zhang37–Reference Rodríguez, Méndez and Torún40,Reference Willet, Sampson and Stampfer42,Reference Yuan, Spiegelman and Rimm43,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45–Reference Jackson, Walker and Cade48,Reference Longnecker, Lissner and Holden51–Reference Chen, Ahsan and Parvez54,Reference Ke, Toshiro and Fengyan57,Reference Sudha, Radhika and Sathya59,Reference Tokudome, Imaeda and Tokudome60,Reference Barrett and Gibson64,Reference Fayet, Flood and Petocz65,Reference Smith, Mitchell and Reay67,Reference Gunes, Imeryuz and Akalin70)} . Another common practice is to convert the correlation coefficients into logarithmic scales. This is done for two main purposes: to improve the normality of the distribution of variables and to simplify the interpretation of correlation values of 0^{(Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45)}. Almost half of the included studies make this modification^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Willett, Reynolds and Cottrell-Hoehner13,Reference Masson, MCNeill and Tomany16,Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Ramón, Micaló and Benítez22,Reference Bonifacj, Gerber and Scali30,Reference Dumartheray, Krieg and Cornuz31,Reference Friis, Kjaer and Stripp35,Reference Dehghan, del Cerro and Zhang37,Reference Rimm, Giovannucci and Stampfer39,Reference Willet, Sampson and Stampfer42,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45–Reference Jackson, Walker and Cade48,Reference Longnecker, Lissner and Holden51–Reference Chen, Ahsan and Parvez54,Reference Ke, Toshiro and Fengyan57,Reference Sudha, Radhika and Sathya59,Reference Tokudome, Imaeda and Tokudome60,Reference Fayet, Flood and Petocz65,Reference Gunes, Imeryuz and Akalin70)} .

In addition to the adjustment for energy, daily variations in the intake of each person (random within-person error) may affect the relationship between the actual and observed nutrient intake. A random within-person error may be due either to real variations in the intake or to intake measurement errors. This type of error tends to attenuate or decrease the regression coefficients between SFFQ and reference methods (dietary records or recalls) towards zero (decrease in the strength of the association)^{(Reference Willett3)}, which is why many authors present de-attenuated correlation analyses based on either unadjusted^{(Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Nöthlings, Hoffmann and Bergmann19,Reference Dehghan, del Cerro and Zhang37,Reference Rodríguez, Méndez and Torún40,Reference Gunes, Imeryuz and Akalin70)} or adjusted coefficients^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Klipstein-Grobusch, Den Breeijen and Goldbohm14,Reference Knudsen, Hatch and Cueto15,Reference Nöthlings, Hoffmann and Bergmann19,Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Bonifacj, Gerber and Scali30,Reference Friis, Kjaer and Stripp35,Reference Rodríguez, Méndez and Torún40,Reference Yuan, Spiegelman and Rimm43,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45,Reference Feskanich, Marshall and Rimm46,Reference Macedo-Ojeda, Vizmanos-Lamotte and Márquez-Sandoval52–Reference Chen, Ahsan and Parvez54,Reference Sudha, Radhika and Sathya59–Reference Yang, Kim and Hwang61,Reference Fayet, Flood and Petocz65,Reference Zack, Irema and Kazonda69)} .

The correlation coefficients used for the validity analyses can vary due to a range of criteria. Regarding the number of items, high variability was found among SFFQ validation studies. However, we did not formally evaluate the correlation between the number of items and validity correlations because of the different populations and the different comparison methods and nutrients that were assessed. The literature associates the more stable reliability of the SFFQ^{(Reference Wakai78)} and higher correlation coefficients^{(Reference Cade, Burley and Warm79)} with their greater number of items, even though this association was not reflected in this review. However, SFFQ with greater numbers of items can be tedious and tiring to answer (which may lead to bias), and having them administered by qualified interviewers may entail a considerable investment of time and money. Ultimately, it is important to keep in mind that the number of items will depend on the purpose of the questionnaire^{(Reference Willett3)}.

About the number of participants, variability between studies and variations in correlation coefficients were also observed. This trend is evident in the study by Nath and Huffman^{(Reference Nath and Huffman38)} in which, probably because of the small number of participants, the coefficient ranges were −0·18 to 0·71 (unadjusted) and −0·34 to 0·55 (adjusted).

Another point of interest regarding SFFQ validation process is the way they are administered. In this review, twenty-two studies state that their questionnaires were administered by an interviewer^{(Reference Rodríguez, Fernández and Pastor1,Reference Moreira, Sampaio and Almeida17,Reference Ramón, Micaló and Benítez22,Reference Rothenberg25,Reference Dehghan, del Cerro and Zhang37,Reference Willet, Sampson and Stampfer42,Reference Sanjeevi, Freeland-Graves and George44,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45,Reference Jackson, Walker and Cade48–Reference Lee, Lee and Wang50,Reference Macedo-Ojeda, Vizmanos-Lamotte and Márquez-Sandoval52–Reference Chen, Ahsan and Parvez54,Reference Jayawardena, Byrne and Soares56,Reference Sudha, Radhika and Sathya59,Reference Yang, Kim and Hwang61–Reference Aoun, Daher and Osta63,Reference Beck, Houston and McNaughton68–Reference Gunes, Imeryuz and Akalin70)} . The advantage of having questionnaires administered by qualified interviewers is the assurance that they will be completed correctly^{(Reference Willett3,Reference Cade, Burley and Warm79)} . When budgetary constraints prohibit hiring specialists to administer questionnaires, self-administration may be a viable alternative. Nonetheless, while self-administration saves costs associated with having someone on hand to explain how to complete the survey, it may entail a greater risk of bias if participants are not adequately informed of the procedures they should follow^{(Reference Cade, Burley and Warm79)}.

Some SFFQ include visual support material to facilitate the estimation of participants’ food intake^{(Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Willett, Reynolds and Cottrell-Hoehner13,Reference Knudsen, Hatch and Cueto15–Reference Moreira, Sampaio and Almeida17,Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Babíc, Sindik and Missoni24,Reference Rothenberg25,Reference Tjønneland, Overvad and Haraldsdóttir27,Reference Van Dongen, Lentjes and Wijckmans28,Reference Fidanza, Gentile and Porrini33,Reference Shatenstein, Nadon and Godin41,Reference Jackson, Walker and Cade48–Reference Lee, Lee and Wang50,Reference Bowen, Bharathi and Kinra53–Reference Jayawardena, Byrne and Soares56,Reference Park, Noh and Song58,Reference Sudha, Radhika and Sathya59,Reference Yang, Kim and Hwang61–Reference Aoun, Daher and Osta63,Reference Horwath66,Reference Zack, Irema and Kazonda69,Reference Gunes, Imeryuz and Akalin70)} . However, this practice has not been clearly associated with higher correlation coefficients. Some studies^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Roddam, Spencer and Banks23,Reference Schröder, Covas and Marrugat26,Reference Van Dongen, Lentjes and Wijckmans28,Reference Dumartheray, Krieg and Cornuz31,Reference Fernández-Ballart, Piñol and Zazpe32,Reference Dehghan, del Cerro and Zhang37,Reference Willet, Sampson and Stampfer42,Reference Yuan, Spiegelman and Rimm43,Reference Jackson, Walker and Cade48–Reference Lee, Lee and Wang50,Reference Macedo-Ojeda, Vizmanos-Lamotte and Márquez-Sandoval52–Reference Bijani, Esmaili and Ghadimi55,Reference Park, Noh and Song58,Reference Sudha, Radhika and Sathya59,Reference Barrett and Gibson64,Reference Beck, Houston and McNaughton68)} mention purposes for which their SFFQ had been previously used, such as epidemiological studies of diabetes mellitus^{(Reference Jackson, Walker and Cade48)} or cancer^{(Reference Martin-Moreno, Boyle and Gorgojo5,Reference Gnardellis, Trichopoulou and Katsouyanni12,Reference Ocké, Bueno-de-Mesquita and Pols20,Reference Willet, Sampson and Stampfer42,Reference Park, Noh and Song58)} .

For validation purposes, in addition to correlation coefficients, Kappa analyses or Bland–Altman plots are usually used. The classification capacities of the tools can be analysed by comparing, through Kappa analyses and contingency tables, the concordance or agreement within the distribution by tertiles, quartiles or quintiles. Results can be reported as an exact agreement (classified in the same category by both methods), plus or minus one category, and gross misclassification^{(Reference Cade, Thompson and Burley76)}. The main advantage of this kind of analysis is that with cross-classification, the percentages misclassified clearly illustrate the likely impact of measurement error. It has been established that 50 % of subjects correctly classified and <10 % of subjects grossly misclassified into thirds, and weighted kappa values above 0·4 are desirable for nutrients of interest^{(Reference Masson, MCNeill and Tomany16)}. However, this is difficult to achieve, since only six studies from the fifteen reporting kappa analyses^{(Reference Rodríguez, Fernández and Pastor1,Reference Masson, MCNeill and Tomany16,Reference Gazan, Vieux and Darmon29,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45,Reference Aoun, Daher and Osta63,Reference Barrett and Gibson64)} report a median or most of subjects with a 50 % or more of correct classification, three report a correct classification between 42 and 49 %, two do not report agreement in percentage^{(Reference Roddam, Spencer and Banks23,Reference Aoun, Daher and Osta63)} and four report a lower level of agreement^{(Reference Smith, Mitchell and Reay67–Reference Gunes, Imeryuz and Akalin70)}.

By the other hand, Bland–Altman charts graphically assess agreement between the methods, displaying the under- or over-estimate of the method to be validated and identifying the possible presence of bias in the estimate. They have the advantage of not being influenced by variations from one person to another^{(Reference Cade, Thompson and Burley76,Reference Serra-Majem, Frost Andersen and Henríque-Sánchez80)} . In total, almost half of the studies reporting Bland–Altman charts (11 of 21) did not show systematic errors for most nutrients^{(Reference Knudsen, Hatch and Cueto15,Reference Moreira, Sampaio and Almeida17,Reference Rothenberg25,Reference Dehghan, del Cerro and Zhang37,Reference Yuan, Spiegelman and Rimm43,Reference Denova-Gutiérrez, Ramírez-Silva and Rodríguez-Ramírez45,Reference Bijani, Esmaili and Ghadimi55,Reference Jayawardena, Byrne and Soares56,Reference Barrett and Gibson64,Reference Fayet, Flood and Petocz65,Reference Beck, Houston and McNaughton68)} and the others confirmed an overestimation of the SFFQ with respect to the reference method.

Other authors have carried out other literature reviews on a SFFQ creation or validation process. For example, Cade et al. ^{(Reference Cade, Thompson and Burley76,Reference Cade, Burley and Warm79)} conducted an electronic database search for English-language papers on the creation, validation and administration of SFFQ from 1980 to 1999, from which they published two papers: a non-systematic review^{(Reference Cade, Thompson and Burley76)} and a semi-systematic review^{(Reference Cade, Burley and Warm79)}. At the end of each paper, they present general recommendations regarding the design, validation and administration of SFFQ. Similarly, Wakai^{(Reference Wakai78)} conducted another literature review to identify articles in which SFFQ were developed and/or validated exclusively for the Japanese population. The main difference between these three studies and ours is that we performed a systematic review using PRISMA statement criteria; moreover, Wakai focused only in studies for Japanese population, while we included all the published studies that met inclusion criteria, regardless of the country.

Based on the analysis of previous proposals^{(Reference Willett3,Reference Cade, Thompson and Burley76,Reference Serra-Majem, Frost Andersen and Henríque-Sánchez80,Reference Dennis, Snetselaar and Nothwehr81)} and our observations, and taking into consideration that the essential reason to validate an instrument is to confirm that it evaluates variables adequately, we recommend taking into consideration specific elements to validate SFFQ that were designed to evaluate global food intake in adults, such as the number of items; the number and sex of participants for validation; administration of the SFFQ (interview or self-reporting); seasonal fluctuations in dietary intake; visual support material for use during questionnaire administration; number of applications of the reference method (dietary records or 24-h recalls); unit of analysis: energy, nutrients and food groups; the use of unadjusted, adjusted and de-attenuated correlation coefficients for the validity analysis (depending on whether the distribution of variables is normal); the Bland–Altman plots; a reproducibility analysis between questionnaire administrations; identification of the statistical package used to perform statistical analyses; and identification of the software used to perform the nutritional analysis.

Finally, we note that when SFFQ validation studies are conducted, validity results are not always favourable for all nutrients or food groups evaluated. Ideally, improvements should be made to these SFFQ, after which they should be revalidated. At least, the limitations of these instruments should be acknowledged, and their results should be interpreted with caution. However, we believe that following the recommendations regarding the limitations of other studies and those we have discussed here will lead to better validation results.

Among the strengths of our study are the inclusion of questionnaires produced in several geographical regions and continents, and that its search period was not limited. Hence, our results include papers that are among the oldest available in the database up to those published as late as January 2020. Furthermore, we have included articles written in English, Portuguese, French and Spanish that were found using systematic search processes. We also extracted data directly from tables to prevent data omission. The scoring system we used is intended to help researchers select those SFFQ that would be the most complete and suitable for the objectives and target populations of their studies.

However, our review also has limitations. One of these is that PubMed was the only search engine used, thus resulting in the exclusion of relevant papers not retrievable through it. We nonetheless decided to only include articles found in PubMed because it is a proven source of articles from highly regarded scientific journals. Also, it may be questionable to have included studies validated only against dietary assessment tools and not studies focused on nutrients for which there might be unbiased biomarkers such as urinary potassium, urinary nitrogen (proteins) and doubly labelled water (energy). However, biomarkers have limitations, which include the fact that recovery markers (those that refer to a measure of absolute intake per 24 h) are not available for most nutrients and may, therefore, provide limited information. Therefore, we consider that biomarkers may be more suitable in studies focused on specific nutrients and not on the general diet, as in our study.

As future perspectives, the creation and validation of new food-frequency consumption questionnaires are justified, since we are living in the personalised nutrition era. Having specific tools for diverse population groups and diverse purposes will support research and application of new knowledge. Besides, it is desirable that these questionnaires not only focus on assessing food but also processed or prepared products, to not limiting existing evidence between food/nutrition and health risks.