Skip to main content

Examples of sex/gender sensitivity in epidemiological research: results of an evaluation of original articles published in JECH 2006–2014

Abstract

Background

During the last decades, sex and gender biases have been identified in various areas of biomedical and public health research, leading to compromised validity of research findings. As a response, methodological requirements were developed but these are rarely translated into research practice. The aim of this study is to provide good practice examples of sex/gender sensitive health research.

Methods

We conducted a systematic search of research articles published in JECH between 2006 and 2014. An instrument was constructed to evaluate sex/gender sensitivity in four stages of the research process (background, study design, statistical analysis, discussion).

Results

In total, 37 articles covering diverse topics were included. Thereof, 22 were evaluated as good practice example in at least one stage; two articles achieved highest ratings across all stages. Good examples of the background referred to available knowledge on sex/gender differences and sex/gender informed theoretical frameworks. Related to the study design, good examples calculated sample sizes to be able to detect sex/gender differences, selected sex/gender sensitive outcome/exposure indicators, or chose different cut-off values for male and female participants. Good examples of statistical analyses used interaction terms with sex/gender or different shapes of the estimated relationship for men and women. Examples of good discussions interpreted their findings related to social and biological explanatory models or questioned the statistical methods used to detect sex/gender differences.

Conclusions

The identified good practice examples may inspire researchers to critically reflect on the relevance of sex/gender issues of their studies and help them to translate methodological recommendations of sex/gender sensitivity into research practice.

Peer Review reports

Background

During the last three decades, sex and gender biases have been identified in biomedical and public health research [13]. Stephenson and McKee wrote, in 1993, that in epidemiology “[g]ender is nearly always treated as a potential confounding variable, the effects of which, if there are any, must be controlled for statistically and then ignored. Gender differences in health per se are seldom given a second glance” [4]. Although some progress has been made, this statement still holds true for large parts of epidemiological research [57].

Sex and gender bias refers to an inadequate representation of men and women in study samples, in research teams, or responsible research positions, as recipients of research funding or as authors of publications [8, 9]. In a broader sense, a comprehensive analysis of biological and social factors associated with being a man or a woman is postulated (sex- and gender-based analysis [10]) to explain or prevent sex- and gender-related health inequalities. Sex and gender bias impinges on the quality of epidemiologic research as it compromises the validity of research findings, leading to inaccurate conclusions about how women (or men) respond to a disease and inadequate treatment decisions [9]. From a feminist social science perspective, Eichler proposed four types of gender bias [11, 12], namely gender insensitivity (ignoring gender aspects), over-generalisation (generalisation of research results to a group that has not been studied), double standard (e.g. by drawing on gender stereotypes for explaining gender differences), and androcentrism (male as the norm). Beery and Zucker [1] used the term ‘sex bias’ referring to both inappropriate representation of men and women and neglected differences in biological mechanisms between males and females. Overall, evidence from different research fields suggests that sex and gender bias leads to gaps in the knowledge base and inequalities in the provision of healthcare – in many cases to the disadvantage of women [13, 14].

A central element of sex/gender sensitive research is the distinction between sex and gender. Although the terms ‘sex’ and ‘gender’ denote different aspects of being male and female, they are often used interchangeably in the literature [1519]. Sex refers to biological characteristics such as chromosomes, hormones or reproductive organs commonly associated with being male or female. Gender refers to sociocultural attributes and related expositions commonly associated with being a man or a woman, such as personality traits, socially constructed roles, access to the labour market, or relative power [17]. However, scholars have warned that the sex–gender dichotomy may mask the complexity, interaction and entanglement between the two concepts [2022]. To avoid simple dualisms between men and women, other scholars have drawn the attention to the within-group variations at the intersections of sex/gender with other social categories, such as socioeconomic position, race/ethnicity, or age [23, 24]. Krieger [25] pointed out that clear conceptual models are needed for considering sex and gender, simultaneously, and their interrelations (‘biologic expressions of gender’, ‘gendered expression of biology’), “Yet, we do not live as a ‘gendered’ person one day and a ‘sexed’ organism the next; we are both, simultaneously, and for any given health outcome, it is an empirical question, not a philosophical principle, as to whether diverse permutations of gender and sex matter – or are irrelevant.” For this purpose, a clear understanding of sex and gender as distinct but entangled concepts and consistent usage of terms are basic requirements of sex/gender sensitive epidemiology [22, 23, 25, 26].

In accordance with other authors [18, 22, 2630], we prefer to use the shortened version ‘sex/gender’ in this article instead of ‘sex and gender’ or ‘sex and/or gender’. By using the slashed version we want to indicate that sex and gender are distinct concepts, while the slash stands for the potential interrelations between biological and sociocultural aspects of being a man, a woman, or a sex/gender diverse person.

To reduce sex/gender bias, concepts such as ‘engendering epidemiology’ [31] or ‘gendered epidemiology’ [24] were proposed and methodological requirements for sex/gender sensitive epidemiological research were developed [13, 2224, 3237]. However, these concepts are rarely translated into research practice [23, 24, 3235, 37]. Yet, paying attention to sex/gender-related aspects has also been implemented as a requirement in research funding schemes as of the European Union [38] and in science publishing [39]. Thus, applying sex/gender sensitive methods is of increasing relevance for health researchers and epidemiologists [34, 36, 37]. In order to increase sex/gender sensitivity of epidemiological research, it might be useful to present good practice examples of sex/gender sensitive health research. We defined good practice examples as sections of published academic articles that have successfully met methodological requirements of sex/gender sensitivity or that have explicitly described sex/gender-related research practices. Good practice examples can inspire researchers aiming to increase sex/gender sensitivity of their studies and help them to translate methodological recommendations of sex/gender sensitivity into research practice [15].

This study was part of the ‘Epi goes Gender’ project in Germany [36] and aimed to identify recent examples of good sex/gender sensitive research practice illustrating how to apply sex/gender sensitive principles in health research. To this end, we evaluated sex/gender-related research articles published in the Journal of Epidemiology and Community Health (JECH). We selected JECH as the basis for our search due to the journal’s scope (social medicine, social epidemiology) and its major contributions to the field of sex/gender sensitive epidemiology, including the supplement entitled ‘Engendering Epidemiology’ in 2007 and further important publications in this field [13, 4042].

Methods

Search strategy

To identify good practice examples of sex/gender sensitive research, we used the following search strategy. Inclusion criteria were (1) published in JECH between 2006 and 2014 to focus on recent development, (2) original research, and (3) title contains ‘sex’ or ‘gender’. The rationale behind this search criterion was to increase the probability to discover articles explicitly dealing with sex/gender aspects [31]. Sex and gender were both included as search terms because we aimed to find examples for biological and sociocultural aspects – and their interrelations. Our aim was not to separate articles dealing with sex from those dealing with gender because, ideally, good practice examples consider both concepts. Articles were excluded if the term ‘sex’ did not denote being male or female or sex-related biological factors but was used in another meaning, e.g. sex(uality), sex work, sex ratio (of newborns). Furthermore, methodological, theoretical and political papers were excluded. The reason for this targeted scope of our search was that we did not aim to provide a comprehensive review of sex/gender sensitivity in epidemiological research, but to illustrate sex/gender sensitivity from purposefully selected examples. Overall, 73 papers were identified that included the terms ‘sex’ or ‘gender’ in their titles. Thereof, 37 articles met all inclusion criteria and were included in the evaluation (Table 1).

Table 1 Number of identified, excluded and included papers

Assessment instrument

We constructed an instrument to evaluate the sex/gender sensitivity of the research presented in the selected articles (Fig. 1 and Additional file 1). Three basic assumptions of sex/gender sensitivity guided the development of the instrument, namely (1) sex/gender-related aspects should be reflected in all stages of the research process [3335, 43], (2) a sound theoretical conceptualisation of sex/gender is necessary, including the complexity, interaction and entanglement between the two concepts as well as within group variations at the intersections of sex/gender with other social categories such as socioeconomic position, ethnicity or age [2325, 42, 4446], and (3) appropriate analytic strategies are to be used [47, 48].

Fig. 1
figure 1

Structure and content of the assessment instrument

Although a clear distinction between sex and gender is important, our focus was more on how biological and sociocultural factors were included and less on the correct and consistent use of the terms sex and gender.

The evaluation of the full texts comprised three steps. In a first step, each main stage of the research process, represented by the sections of the article (background, study design, statistical analysis, discussion), was screened for addressing any sex/gender-related aspects. In a second step, specific attention was drawn to the operationalisation of sex/gender beyond the binary male/female category. The third step included an in-depth examination of the identified sex/gender-related aspects and a classification of each section into one of three categories: ‘good practice example’, ‘intermediate’, ‘neither a good practice example nor intermediate’. Article sections were classified as ‘good practice example’ if it became apparent that sex/gender concepts guided the related stage of the research process. The category ‘intermediate’ was chosen if only some sex/gender aspects were addressed. The article sections were classified as ‘neither a good practice example nor intermediate’ if sex/gender differences or similarities were not addressed or addressed without any further justification, for example, the sole presentation of sex/gender stratified results. To guide the evaluation, the concept of diagnostic questions [43] was applied. The diagnostic questions were mainly derived based on Eichler et al. [43] and Hammarström [49]. Drafts of the assessment instrument were revised after consultation of experts qualified in epidemiology, public health and gender research. The final version was pretested in a multidisciplinary group of associates of the project ‘Epi goes Gender’ in 2012. The assessment instrument was a generic one and thus not specific to any field of epidemiologic research or study design. It should be highlighted that it was not our aim to assess the quality of the articles in terms of their contributions to their specific fields of research.

Evaluation

To achieve intersubjective validity, articles were independently evaluated by two evaluators. The first evaluation was completed by CB or FG, both biometricians and trained on the job in principles of sex/gender sensitive health research. The second evaluation was completed by IJ, a social scientist working in the field of sex/gender sensitive health research as a senior researcher and principal investigator of the ‘Epi goes Gender’ project. The instrument was filled in based on the full text without considering any additional information such as design papers or supplementary material. In case of discordant evaluations, a consensus was achieved by communicative validation. In addition to the evaluation, contents of the good practice examples were extracted to illustrate how sex/gender sensitivity was achieved.

Results

We will first give a rough summary of the results and will then describe good practice examples identified for the different sections of the articles. The topics of the included articles were quite diverse as can be retrieved from article titles [5086].

In 32 out of 37 articles, sex/gender-related aspects were explicitly named among the research aims, which can be grouped as follows: (1) investigation of the relationships between gender (inequalities) and health [50, 58, 62, 67, 69, 75]; (2) analysis of sex and/or gender differences [51, 52, 5456, 59, 66, 68, 70, 71, 78, 83, 86]; (3) effect modification analysis by sex/gender [53, 61, 76]; (4) investigation of sex/gender-based origins of health outcomes [57, 60, 63, 73, 77, 80, 81, 84, 85]; and (5) exploration of gender bias [82].

Results of the screening (steps 1 and 2)

Sex/gender aspects were addressed at least to some extent in 32 out of 37 background sections and in 31 out of 37 discussion sections. In the study design section, 19 out of 37 articles addressed sex/gender aspects. In all included articles, statistical analyses were adjusted for or stratified by sex/gender.

In the second step, we analysed whether sex and gender were operationalised beyond a binary male/female category, i.e. whether more than a single indicator for being male or female was used such as societal indicators of gender relations (e.g. gender equality index) or biological markers (e.g. hormone status). This was the case in 11 articles [51, 53, 58, 63, 6671, 83] for gender and in 5 articles [54, 55, 60, 62, 81] for both sex and gender. In 21 articles [50, 52, 56, 57, 59, 61, 64, 65, 7280, 82, 8486] sex/gender was operationalised as male/female only.

Results of the comprehensive assessment (step 3)

Twenty-two out of 37 articles were rated as good practice example in at least one section and 2 articles achieved highest ratings across all four sections (Table 2).

Table 2 Assessment of sex/gender sensitivity in each section of the selected articles – Synopsis of the results

In 14 out of 37 articles at least one section was classified as intermediate category. Only one article was rated as ‘neither a good practice example nor intermediate’ in each section. The proportion of good practice examples was highest for the background (17/37) and discussion (14/37) sections. In the study design and the statistical analysis sections, 6 out of 37 and 5 out of 37 articles were identified as good practice examples, respectively.

Background section

All 17 good practice examples for this section referred to available knowledge on sex/gender differences. Some authors referred to gender-informed theoretical frameworks or explanatory models such as the multiple role model, role expansion theory [51, 67], homemaker hypotheses [61], gendered socialisation in cross-cultural comparisons [62], societal gender equality [66], or biological and social explanatory hypotheses for neonatal mortality [81]. The investigation of Heys et al. [60] was explicitly driven by biological differences leading to sex specific effects on lipids and fat patterning. Some authors critically reflected on the used research methods, e.g. averaging effect estimates across subgroups and thereby masking potential effect modifications [63, 68].

Study design section

Good practice examples for this section included the following elements: (1) selection of outcome variables that reflect sex/gender differences, (2) calculation of sample sizes enabling the detection of sex/gender differences, (3) selection of sex/gender sensitive exposure indicators, or (4) choosing specific cut-off values for exposure/outcome classifications for male and female participants. For example, Rosenstock et al. [81] applied sex- and gender-based operationalisations of the outcome variable and of influencing factors referring to the entanglement concept [22]. Heys et al. [60] selected the outcome variables based on biological mechanisms concerning the role of sex steroids and growth hormones during puberty. Ruiz-Cantero et al. [82] conducted a pilot study to obtain prevalences for their sample size calculation to attain sufficient statistical power for the detection of differences between men and women in the main study. Harryson et al. [58] as well as Mansdotter et al. [67] selected their exposure variable (work in the domestic and public sphere) based on concepts of gender equality and constructed a gender equality index. Escribà-Agüir et al. [54] and Escribà-Agüir and Artazcoz [55] used different cut off-values for men and women of the Edinburgh Postnatal Depression Scale as prior research had indicated that the threshold for men is two scale points lower compared to women.

Statistical analysis section

Good practice examples for this section explicitly accounted for sex/gender aspects in the statistical analyses. This included, for example, the derivation of specific measures accounting for sex/gender differences in exposure or outcome variables [67]. Models were not only stratified by or adjusted for the sex/gender variable (male/female), but further included, for example, interactions with biological or social factors to disentangle sex/gender differences [63, 81]. Conducting a sex/gender sensitive analysis could also mean that different models were fitted to unravel the effects of biological, social and environmental factors [68, 81]. In one example, the shape of the estimated relationship between alcohol consumption and neighbourhood deprivation was allowed to be non-linear and to vary between men and women [69].

Discussion section

Good practice examples for the discussion section provided interpretations of their findings by drawing on possible sex/gender-related explanatory approaches or theoretical models [51, 6668, 81, 83]. Likewise, methodological issues were addressed [63, 67, 75]. For example, in their analysis of neighbourhood deprivation effects on health, Kavanagh et al. [63] critically reflected on the statistical methods used to detect differences between men and women and compared stratified analyses with analyses using statistical interactions. While stratified analyses separately assess the effects among women and men, statistical interactions indicate differences in the effects between women and men. This example highlights the clarity about the question whether effects of exposures in separated groups or differences of effect sizes across groups are to be determined that is needed for the definition of research questions and for subsequent statistical modelling.

Discussion

This study provides a collection of good practice examples of sex/gender sensitivity based on an evaluation of sex/gender-related health research in a selected scientific journal. Good practice examples were identified in four article sections/stages of the research process and in studies with different aims and designs. The results include examples for many methodological issues that have been discussed in the context of sex/gender sensitive research [23, 24, 3234, 36, 37]. More good practice examples were found in the background and discussion sections than in the study design and statistical analysis sections. It is interesting to note that only one example (in two articles) was identified where different cut off-values for men and women were used [54, 55]; one other example conducted a pilot study to obtain prevalences in males and females for a subsequent sample size calculation [82].

To the authors’ knowledge this is the first study aiming to identify examples of good practice of sex/gender sensitive research in epidemiology and thus providing practical ideas of sex/gender sensitivity for researchers based on existing science. The presentation of good practice examples demonstrates how sex/gender sensitivity can be accomplished. This may encourage researchers and reviewers to carefully reflect on their research and reviewing practice and could enhance the inclusion of sex/gender aspects in future research.

Our approach can be located in the context of the ‘Gendered Innovations Framework’ [46]. In accordance with this framework, our aim was to illustrate how concepts such as sex and gender can be integrated into the different stages of the research process [46]. The ‘Gendered Innovations’ website [87] offers information for sex/gender sensitive methodological principles in biomedical and public health research and has recently been adopted by the EU Horizon 2020 funding scheme [88]. Currently, ‘Gendered Innovations’ provides seven case studies from medicine and public health illustrating challenges of sex/gender sensitive research, such as the identification of relevant sex/gender factors and their interactions in colorectal cancer or a critical sex/gender analysis of a dietary assessment using a food frequency questionnaire.

New guidelines to advance sex/gender reporting by the European Association of Science Editors are currently under way [39]. In this context ‘sex and gender questions’ were recently published [89], which can be used as a checklist when planning a study. There is a substantial overlap between these questions and the diagnostic questions we used in our assessment instrument. Furthermore, the questions emphasise that both the consideration or not of sex/gender aspects should be justified.

The research topics of the included articles were quite diverse. The evaluators’ expertise covered several research fields but not all topics of the included articles. As a consequence our approach was a generic one, focusing on sex/gender-related aspects but not evaluating the quality of the articles with respect to the state of the art in the specific research field. However, this generic approach enables us to demonstrate that high-quality sex/gender sensitive research can be conducted in a broad area of health research.

Based on the diagnostic questions of our assessment instrument and our findings in the evaluated articles, we developed a checklist of practical steps towards sex/gender sensitivity across the stages of the research process (Table 3). In the right column of Table 3 the steps are illustrated by examples from one research article that achieved high ratings across all four sections. The practical steps do not cover all aspects that have been discussed in the field of sex/gender sensitive research, but can also be used in combination with other concepts or guidelines [22, 39, 9093].

Table 3 Illustrated checklist of practical steps of sex/gender sensitivity in the stages of the research process

Several measures were taken to establish intersubjective validity such as diagnostic questions to provide an orientation, double ratings and consensus building strategies. Nevertheless, the professional background of the evaluators may have had some influences on the assessment, for example, assigning the operationalisation of the variables to the statistical modelling part or to the study design. Searching for sex or gender in titles only may be viewed as a limited approach. However, comparing this strategy with an extended search including the abstracts of the articles revealed that the strategy was very successful in identifying studies that explicitly aim to investigate sex/gender-related aspects [94]. A further limitation could be that we did not examine supplementary materials or related papers of the included studies. These materials may have contained more information on aspects of the study design such as the validity of the measures and sample size calculations. However, our expectation was that the key elements of the study design would be published in the original article.

Conclusion

This study provides good practice examples of sex/gender sensitivity in epidemiological research. The examples include different lines of research, such as questions of gender research in epidemiology, e.g. influences of individual or societal gender relations on health, as well as questions of the relevance of sex/gender aspects for the definition of exposures, outcomes and respective pathways (sex/gender-based analysis). While gender research presupposes a specific epistemological perspective and research interest which may not be shared by many epidemiologists, sex/gender-based analysis is relevant to the quality of epidemiological research as it contributes to our understanding of causal pathways. Determining the relevance and well-founded inclusion or exclusion of sex/gender aspects and the careful epidemiological analysis is an important requirement for a more equitable provision of healthcare and preferably gender transformative [95] preventative services for women and men.

Measures to motivate and enable researchers to increase sex/gender sensitivity should focus on aspects of the study design and the statistical analysis, e.g. sex/gender sensitive selection of determinants and outcomes, operationalisation of sex/gender aspects, and power calculation.

Next steps in the analysis of sex/gender sensitivity in epidemiological research would include the coverage of a broader range of epidemiological journals extracting both positive examples as well as examples of sex/gender bias. This should also include an assessment of the consistency and conceptual clarity of the terms sex and gender [96].

What is already known?

  • Epidemiological research is often of limited significance in terms of sex/gender sensitivity.

  • There is a growing body of knowledge concerning the analysis and avoidance of sex/gender bias.

What this paper adds?

This paper serves as a first step to use positive examples in order to develop and disseminate sex/gender sensitive research and provides indications on how to practically implement sex/gender aspects.

Abbreviations

JECH:

Journal of Epidemiology and Community Health

References

  1. Beery AK, Zucker I. Sex bias in neuroscience and biomedical research. Neurosci Biobehav Rev. 2011;35:565–72. doi:10.1016/j.neubiorev.2010.07.002.

    Article  PubMed  Google Scholar 

  2. Gelb K, Pederson A, Greaves L. How have health promotion frameworks considered gender? Health Promot Int. 2012;27:445–52. doi:10.1093/heapro/dar087.

    Article  PubMed  Google Scholar 

  3. Zahm SH, Pottern LM, Lewis DR, Ward MH, White DW. Inclusion of women and minorities in occupational cancer epidemiologic research. J Occup Environ Med. 1994;36:842–7.

    Article  CAS  Google Scholar 

  4. Stephenson P, McKee M. Look twice. Eur J Public Health. 1993;3:151–2.

    Article  Google Scholar 

  5. Hohenadel K, Raj P, Demers PA, Zahm SH, Blair A. The inclusion of women in studies of occupational cancer: a review of the epidemiologic literature from 1991-2009. Am J Ind Med. 2015;58:276–81. doi:10.1002/ajim.22424.

    Article  PubMed  Google Scholar 

  6. Mazure CM, Jones DP. Twenty years and still counting: including women as participants and studying sex and gender in biomedical research. BMC Womens Health. 2015;15:94. doi:10.1186/s12905-015-0251-9.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Bolte G. Gender in epidemiology. State of discussion and perspectives. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2008;51:3–12. doi:10.1007/s00103-008-0414-z.

    Article  CAS  PubMed  Google Scholar 

  8. Garcia-Calvente Mdel M, Ruiz-Cantero MT, Del Rio-Lozano M, Borrell C, Lopez-Sancho MP. Gender inequalities in research in public health and epidemiology in Spain (2007-2014). Gac Sanit. 2015;29:404–11. doi:10.1016/j.gaceta.2015.07.013.

    Article  PubMed  Google Scholar 

  9. Ovseiko PV, Greenhalgh T, Adam P, Grant J, Hinrichs-Krapels S, Graham KE, Valentine PA, Sued O, Boukhris OF, Al Olaqi NM, et al. A global call for action to include gender in research impact assessment. Health Res Policy Syst. 2016;14:50. doi:10.1186/s12961-016-0126-z.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Sex and Gender-Based Analysis - Health Canada. http://www.hc-sc.gc.ca/hl-vs/gender-genre/analys/index-eng.php. Accessed 1 April 2016.

  11. Nonsexist EM, Methods R. A practical guide. Boston: Allen and Unwin; 1988.

    Google Scholar 

  12. Eichler M. Nonsexist Research Methods. A Practical Guide. Hoboken: Taylor and Francis; 2013. Online-Resource.

    Google Scholar 

  13. Ruiz MT, Verbrugge LM. A two way view of gender bias in medicine. J Epidemiol Community Health. 1997;51:106–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Ruiz-Cantero MT, Vives-Cases C, Artazcoz L, Delgado A, Garcia Calvente MM, Miqueo C, Montero I, Ortiz R, Ronda E, Ruiz I, Valls C. A framework to analyse gender bias in epidemiological research. J Epidemiol Community Health. 2007;61:ii46–53. doi:10.1136/jech.2007.062034.

    PubMed  PubMed Central  Google Scholar 

  15. Gahagan J, Gray K, Whynacht A. Sex and gender matter in health research: addressing health inequities in health research reporting. Int J Equity Health. 2015;14:12. doi:10.1186/s12939-015-0144-4.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Hammarström A, Annandale E. A conceptual muddle: an empirical analysis of the use of 'sex' and 'gender' in 'gender-specific medicine' journals. PLoS One. 2012;7:e34193. doi:10.1371/journal.pone.0034193.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Ritz SA, Antle DM, Cote J, Deroy K, Fraleigh N, Messing K, Parent L, St-Pierre J, Vaillancourt C, Mergler D. First steps for integrating sex and gender considerations into basic experimental biomedical research. FASEB J. 2014;28:4–13. doi:10.1096/fj.13-233395.

    Article  CAS  PubMed  Google Scholar 

  18. Westbrook L, Saperstein A. New categories are not enough rethinking the measurement of sex and gender in social surveys. Gender Soc. 2015;29:534–60.

    Article  Google Scholar 

  19. Clayton J, Tannenbaum C. Reporting sex, gender, or both in clinical research? JAMA. 2016;316:1863–4. doi:10.1001/jama.2016.16405.

    Article  PubMed  Google Scholar 

  20. Fausto‐Sterling A. The bare bones of sex: part 1 – sex and gender. Signs. 2005;30:1491–527.

    Article  Google Scholar 

  21. Short SE, Yang YC, Jenkins TM. Sex, gender, genetics, and health. Am J Public Health. 2013;103 Suppl 1:S93–S101. doi:10.2105/AJPH.2013.301229.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Springer KW, Mager Stellman J, Jordan-Young RM. Beyond a catalogue of differences: a theoretical frame and good practice guidelines for researching sex/gender in human health. Soc Sci Med. 2012;74:1817–24. doi:10.1016/j.socscimed.2011.05.033.

    Article  PubMed  Google Scholar 

  23. Doyal L. Sex and gender: the challenges for epidemiologists. Int J Health Serv. 2003;33:569–79.

    Article  PubMed  Google Scholar 

  24. White A, Richardson N. Gendered epidemiology: making men's health visible in epidemiological research. Public Health. 2011;125:407–10. doi:10.1016/j.puhe.2011.04.012.

    Article  CAS  PubMed  Google Scholar 

  25. Krieger N. Genders, sexes, and health: what are the connections-and why does it matter? Int J Epidemiol. 2003;32:652–7. doi:10.1093/ije/dyg156.

    Article  PubMed  Google Scholar 

  26. Fishman JR, Wick JG, Koenig BA. The use of “sex” and “gender” to define and characterize meaningful differences between men and women. Agenda for research on women’s health for the 21st century. Washington: Department of Health and Human Services; 1999.

    Google Scholar 

  27. Krieger N, Fee E. Man-made medicine and women's health: the biopolitics of sex/gender and race/ethnicity. Int J Health Serv. 1994;24:265–83.

    Article  CAS  PubMed  Google Scholar 

  28. Doull M, Welch V, Puil L, Runnels V, Coen SE, Shea B, O'Neill J, Borkhoff C, Tudiver S, Boscoe M. Development and evaluation of 'briefing notes' as a novel knowledge translation tool to aid the implementation of sex/gender analysis in systematic reviews: a pilot study. PLoS One. 2014;9:e110786. doi:10.1371/journal.pone.0110786.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Runnels V, Tudiver S, Doull M, Boscoe M. The challenges of including sex/gender analysis in systematic reviews: a qualitative survey. Syst Rev. 2014;3:33. doi:10.1186/2046-4053-3-33.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Jordan-Young R, Rumiati RI. Hardwired for sexism? Approaches to sex/gender in neuroscience. Neuroethics. 2012;5:305–15. doi:10.1007/s12152-011-9134-4.

    Article  Google Scholar 

  31. Garcia AM, Bartley M, Alvarez-Dardet C. Engendering epidemiology. J Epidemiol Community Health. 2007;61:ii1–2. doi:10.1136/jech.2007.069658.

    PubMed  PubMed Central  Google Scholar 

  32. Moerman CJ, van Mens-Verhulst J. Gender-sensitive epidemiological research: suggestions for a gender-sensitive approach towards problem definition, data collection and analysis in epidemiological research. Psychol Health Med. 2004;9:41–52. doi:10.1080/13548500310001637742.

    Article  Google Scholar 

  33. Artazcoz L, Borrell C, Cortes I, Escriba-Aguir V, Cascant L. Occupational epidemiology and work related inequalities in health: a gender perspective for two complementary approaches to work and health research. J Epidemiol Community Health. 2007;61:ii39–45. doi:10.1136/jech.2007.059774.

    PubMed  PubMed Central  Google Scholar 

  34. Nieuwenhoven L, Klinge I. Scientific excellence in applying sex- and gender-sensitive methods in biomedical and health research. J Womens Health (Larchmt). 2010;19:313–21. doi:10.1089/jwh.2008.1156.

    Article  Google Scholar 

  35. Clougherty JE. A growing role for gender analysis in air pollution epidemiology. Cien Saude Colet. 2011;16:2221–38.

    Article  PubMed  Google Scholar 

  36. Jahn I, Gansefort D, Kindler-Röhrborn A, Pfleiderer B. Sex- and gender-sensitive research in epidemiology and medicine: how can this be achieved? Aims and first results of the network "Sex-/Gender-Sensitive Research in Epidemiology, Neurosciences and Genetics/Cancer Research". Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2014;57:1038–46. doi:10.1007/s00103-014-2010-8.

    Article  CAS  PubMed  Google Scholar 

  37. Jahn I. Methodische Probleme einer geschlechtergerechten Gesundheitsforschung. In: Kolip P, Hurrelmann K, editors. Handbuch Geschlecht und Gesundheit Männer und Frauen im Vergleich, vollst. überarb. und erw. Aufl. 2nd ed. Bern: Hogrefe; 2016. p. 71–86.

    Google Scholar 

  38. European Commission. Fact Sheet: Gender Equality in Horizon 2020. 2013. https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/FactSheet_Gender_2.pdf. Accessed 14 Jun 2016.

    Google Scholar 

  39. Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and gender equity in research: rationale for the SAGER guidelines and recommended use. Res Integr Peer Rev. 2016;1:1–9. doi:10.1186/s41073-016-0007-6.

    Article  Google Scholar 

  40. Hammarström A, Johansson K, Annandale E, Ahlgren C, Aléx L, Christianson M, Elwér S, Eriksson C, Fjellman-Wiklund A, Gilenstam K. Central gender theoretical concepts in health research: the state of the art. J Epidemiol Community Health. 2014;68:185–90.

    Article  PubMed  Google Scholar 

  41. Jiménez-Rodrigo ML, Martínez-Morante E, del Mar G-CM, Álvarez-Dardet C. Through gender parity in scientific publications. J Epidemiol Community Health. 2008;62:474–5.

    Article  PubMed  Google Scholar 

  42. Phillips SP. Measuring the health effects of gender. J Epidemiol Community Health. 2008;62:368–71. doi:10.1136/jech.2007.062158.

    Article  CAS  PubMed  Google Scholar 

  43. Eichler M, Fuchs J, Maschewsky-Schneider U. Richtlinien zur Vermeidung von Gender Bias in der Gesundheitsforschung. Zeitschrift für Gesundheitswissenschaften. 2000;8:293–310.

    Google Scholar 

  44. Phillips SP. Defining and measuring gender: a social determinant of health whose time has come. Int J Equity Health. 2005;4:11.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Kennedy SM, Koehoorn M. Exposure assessment in epidemiology: does gender matter? Am J Ind Med. 2003;44:576–83.

    Article  PubMed  Google Scholar 

  46. Singh S, Klinge I. Mining for methods - A critical review of the theoretical and methodological contributions of feminist science scholars for biomedicine and public health research. Zeitschrift für Geschlechterstudien. 2015;21:15–31.

    Article  Google Scholar 

  47. Kunkel SR, Atchley RC. Why gender matters: being female is not the same as not being male. Am J Prev Med. 1996;12:294.

    CAS  PubMed  Google Scholar 

  48. Jahn I, Foraita R. Geschlechtergerechte epidemiologische Datenanalyse: Methodische Aspekte und empirische Befunde. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2008;51:13–27.

    Article  CAS  PubMed  Google Scholar 

  49. Hammarström A. A tool for developing gender research in medicine: examples from the medical literature on work life. Gend Med. 2007;4:S123–32.

    Article  PubMed  Google Scholar 

  50. Bambra C, Pope D, Swami V, Stanistreet D, Roskam A, Kunst A, Scott-Samuel A. Gender, health inequalities and welfare state regimes: a cross-national study of 13 European countries. J Epidemiol Community Health. 2009;63:38–44. doi:10.1136/jech.2007.070292.

    Article  CAS  PubMed  Google Scholar 

  51. Berntsson L, Lundberg U, Krantz G. Gender differences in work-home interplay and symptom perception among Swedish white-collar employees. J Epidemiol Community Health. 2006;60:1070–6. doi:10.1136/jech.2005.042192.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Boone-Heinonen J, Gordon-Larsen P. Life stage and sex specificity in relationships between the built and socioeconomic environments and physical activity. J Epidemiol Community Health. 2011;65:847–52. doi:10.1136/jech.2009.105064.

    Article  PubMed  Google Scholar 

  53. Borrell C, Muntaner C, Sola J, Artazcoz L, Puigpinos R, Benach J, Noh S. Immigration and self-reported health status by social class and gender: the importance of material deprivation, work organisation and household labour. J Epidemiol Community Health. 2008;62:e7.

    Article  CAS  PubMed  Google Scholar 

  54. Escribà-Agüir V, Gonzalez-Galarzo MC, Barona-Vilar C, Artazcoz L. Factors related to depression during pregnancy: are there gender differences? J Epidemiol Community Health. 2008;62:410–4. doi:10.1136/jech.2007.063016.

    Article  Google Scholar 

  55. Escribà-Agüir V, Artazcoz L. Gender differences in postpartum depression: a longitudinal cohort study. J Epidemiol Community Health. 2011;65:320–6. doi:10.1136/jech.2008.085894.

    Article  PubMed  Google Scholar 

  56. Gissler M, Rahkonen O, Mortensen L, Arntzen A, Cnattingius S, Nybo Andersen AM, Hemminki E. Sex differences in child and adolescent mortality by parental education in the Nordic countries. J Epidemiol Community Health. 2012;66:57–63. doi:10.1136/jech.2009.093153.

    Article  PubMed  Google Scholar 

  57. Haukenes I, Riise T, Haug K, Farbu E, Maeland JG. Smokers' increased risk for disability pension: social confounding or health-mediated effects? Gender-specific analyses of the Hordaland Health Study cohort. J Epidemiol Community Health. 2013;67:758–64. doi:10.1136/jech-2012-202182.

    Article  PubMed  Google Scholar 

  58. Harryson L, Novo M, Hammarström A. Is gender inequality in the domestic sphere associated with psychological distress among women and men? Results from the Northern Swedish Cohort. J Epidemiol Community Health. 2012;66:271–6. doi:10.1136/jech.2010.109231.

    Article  PubMed  Google Scholar 

  59. Hernandez DC, Pressler E. Accumulation of childhood poverty on young adult overweight or obese status: race/ethnicity and gender disparities. J Epidemiol Community Health. 2014;68:478–84. doi:10.1136/jech-2013-203062.

    Article  PubMed  Google Scholar 

  60. Heys M, Jiang C, Cheng KK, Zhang W, Lam TH, Leung GM, Schooling CM. Does childhood meat eating contribute to sex differences in risk factors for ischaemic heart disease in a developing population? J Epidemiol Community Health. 2011;65:522–8. doi:10.1136/jech.2009.099143.

    Article  PubMed  Google Scholar 

  61. Hollander AC, Bruce D, Ekberg J, Burstrom B, Ekblad S. Hospitalisation for depressive disorder following unemployment - differentials by gender and immigrant status: a population-based cohort study in Sweden. J Epidemiol Community Health. 2013;67:875–81. doi:10.1136/jech-2013-202701.

    Article  PubMed  Google Scholar 

  62. Ikeda A, Kawachi I, Iso H, Inoue M, Tsugane S. Gender difference in the association between social support and metabolic syndrome in Japan: the 'enkai' effect? J Epidemiol Community Health. 2011;65:71–7. doi:10.1136/jech.2009.090613.

    Article  PubMed  Google Scholar 

  63. Kavanagh AM, Bentley R, Turrell G, Broom DH, Subramanian SV. Does gender modify associations between self rated health and the social and economic characteristics of local environments? J Epidemiol Community Health. 2006;60:490–5. doi:10.1136/jech.2005.043562.

    Article  PubMed  PubMed Central  Google Scholar 

  64. King WD, Minor P, Ramirez Kitchen C, Ore LE, Shoptaw S, Victorianne GD, Rust G. Racial, gender and geographic disparities of antiretroviral treatment among US Medicaid enrolees in 1998. J Epidemiol Community Health. 2008;62:798–803. doi:10.1136/jech.2005.045567.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Kolarcik P, Geckova AM, Orosova O, van Dijk JP, Reijneveld SA. Predictors of health-endangering behaviour among Roma and non-Roma adolescents in Slovakia by gender. J Epidemiol Community Health. 2010;64:1043–8. doi:10.1136/jech.2009.092098.

    Article  CAS  PubMed  Google Scholar 

  66. Kovess-Masfety V, Boyd A, van de Velde S, de Graaf R, Vilagut G, Haro JM, Florescu S, O'Neill S, Weinberg L, Alonso J. Are there gender differences in service use for mental disorders across countries in the European Union? Results from the EU-World Mental Health survey. J Epidemiol Community Health. 2014;68:649–56. doi:10.1136/jech-2013-202962.

    Article  PubMed  Google Scholar 

  67. Mansdotter A, Lindholm L, Lundberg M, Winkvist A, Ohman A. Parental share in public and domestic spheres: a population study on gender equality, death, and sickness. J Epidemiol Community Health. 2006;60:616–20. doi:10.1136/jech.2005.041327.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Matheson FI, White HL, Moineddin R, Dunn JR, Glazier RH. Neighbourhood chronic stress and gender inequalities in hypertension among Canadian adults: a multilevel analysis. J Epidemiol Community Health. 2010;64:705–13. doi:10.1136/jech.2008.083303.

    Article  PubMed  Google Scholar 

  69. Matheson FI, White HL, Moineddin R, Dunn JR, Glazier RH. Drinking in context: the influence of gender and neighbourhood deprivation on alcohol consumption. J Epidemiol Community Health. 2012;66:e4. doi:10.1136/jech.2010.112441.

    Article  PubMed  Google Scholar 

  70. Matheson FI, Smith KL, Moineddin R, Dunn JR, Glazier RH. Mental health status and gender as risk factors for onset of physical illness over 10 years. J Epidemiol Community Health. 2014;68:64–70. doi:10.1136/jech-2013-202838.

    Article  PubMed  Google Scholar 

  71. Matheson FI, Smith KL, Fazli GS, Moineddin R, Dunn JR, Glazier RH. Physical health and gender as risk factors for usage of services for mental illness. J Epidemiol Community Health. 2014;68:971–8. doi:10.1136/jech-2014-203844.

    Article  PubMed  PubMed Central  Google Scholar 

  72. McCormack GR, Friedenreich C, Shiell A, Giles-Corti B, Doyle-Baker PK. Sex- and age-specific seasonal variations in physical activity among adults. J Epidemiol Community Health. 2010;64:1010–6. doi:10.1136/jech.2009.092841.

    Article  CAS  PubMed  Google Scholar 

  73. Milner A, Page A, LaMontagne AD. Duration of unemployment and suicide in Australia over the period 1985-2006: an ecological investigation by sex and age during rising versus declining national unemployment rates. J Epidemiol Community Health. 2013;67:237–44. doi:10.1136/jech-2012-201594.

    Article  PubMed  Google Scholar 

  74. Mindell JS, Knott CS, Ng Fat LS, Roth MA, Manor O, Soskolne V, Daoud N. Explanatory factors for health inequalities across different ethnic and gender groups: data from a national survey in England. J Epidemiol Community Health. 2014;68:1133–44. doi:10.1136/jech-2014-203927.

    Article  CAS  PubMed  Google Scholar 

  75. Nante N, Messina G, Cecchini M, Bertetto O, Moirano F, McKee M. Sex differences in use of interventional cardiology persist after risk adjustment. J Epidemiol Community Health. 2009;63:203–8. doi:10.1136/jech.2008.077537.

    Article  CAS  PubMed  Google Scholar 

  76. Niclasen J, Nybo Andersen AM, Teasdale TW, Strandberg-Larsen K. Prenatal exposure to alcohol, and gender differences on child mental health at age seven years. J Epidemiol Community Health. 2014;68:224–32. doi:10.1136/jech-2013-202956.

    Article  PubMed  Google Scholar 

  77. Pitel L, Madarasova Geckova A, Kolarcik P, Halama P, Reijneveld SA, van Dijk JP. Gender differences in the relationship between religiosity and health-related behaviour among adolescents. J Epidemiol Community Health. 2012;66:1122–8. doi:10.1136/jech-2011-200914.

    Article  PubMed  Google Scholar 

  78. Ratner PA, Tzianetas R, Tu AW, Johnson JL, Mackay M, Buller CE, Rowlands M, Reime B. Myocardial infarction symptom recognition by the lay public: the role of gender and ethnicity. J Epidemiol Community Health. 2006;60:606–15. doi:10.1136/jech.2005.037952.

    Article  PubMed  PubMed Central  Google Scholar 

  79. Regidor E, de Mateo S, Ronda E, Sanchez-Paya J, Gutierrez-Fisac JL, de la Fuente L, Pascual C. Heterogeneous trend in smoking prevalence by sex and age group following the implementation of a national smoke-free law. J Epidemiol Community Health. 2011;65:702–8. doi:10.1136/jech.2009.091181.

    Article  CAS  PubMed  Google Scholar 

  80. Rigby JE, Dorling D. Mortality in relation to sex in the affluent world. J Epidemiol Community Health. 2007;61:159–64. doi:10.1136/jech.2006.047381.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Rosenstock S, Katz J, Mullany LC, Khatry SK, LeClerq SC, Darmstadt GL, Tielsch JM. Sex differences in neonatal mortality in Sarlahi, Nepal: the role of biology and environment. J Epidemiol Community Health. 2013;67:986–91. doi:10.1136/jech-2013-202646.

    Article  PubMed  Google Scholar 

  82. Ruiz-Cantero MT, Ronda E, Alvarez-Dardet C. The importance of study design strategies in gender bias research: the case of respiratory disease management in primary care. J Epidemiol Community Health. 2007;61:ii11–6. doi:10.1136/jech.2007.060301.

    PubMed  PubMed Central  Google Scholar 

  83. Staehelin K, Schindler C, Spoerri A, Zemp SE. Marital status, living arrangement and mortality: does the association vary by gender? J Epidemiol Community Health. 2012;66:e22. doi:10.1136/jech.2010.128397.

    Article  PubMed  Google Scholar 

  84. Strand BH, Murray ET, Guralnik J, Hardy R, Kuh D. Childhood social class and adult adiposity and blood-pressure trajectories 36-53 years: gender-specific results from a British birth cohort. J Epidemiol Community Health. 2012;66:512–8. doi:10.1136/jech.2010.115220.

    Article  PubMed  Google Scholar 

  85. Värnik A, Kolves K, van der Feltz-Cornelis CM, Marusic A, Oskarsson H, Palmer A, Reisch T, Scheerder G, Arensman E, Aromaa E, et al. Suicide methods in Europe: a gender-specific analysis of countries participating in the "European Alliance Against Depression". J Epidemiol Community Health. 2008;62:545–51. doi:10.1136/jech.2007.065391.

    Article  PubMed  PubMed Central  Google Scholar 

  86. Vigna-Taglianti F, Vadrucci S, Faggiano F, Burkhart G, Siliquini R, Galanti MR. Is universal prevention against youths' substance misuse really universal? Gender-specific effects in the EU-Dap school-based prevention trial. J Epidemiol Community Health. 2009;63:722–8. doi:10.1136/jech.2008.081513.

    Article  CAS  PubMed  Google Scholar 

  87. European Commission. Research & Innovation. http://ec.europa.eu/research/swafs/gendered-innovations/index_en.cfm?pg=home. Accessed June 13 2016.

  88. European Commission. Gendered Innovations - How Gender Analysis Contributes to Research. Luxembourg: Publications Office of the European Union; 2013. p. 137.

    Google Scholar 

  89. European Association of Science Editors. EASE Toolkit for Authors: Sex and Gender Questions. 2015. http://www.ease.org.uk/publications/sex-and-gender-questions/. Accessed Aug 17 2016.

  90. European Commission. Research and Innovation. Gendered Innovations in Science, Health & Medicine, Engineering and Environment. Health and Medicine Checklist. http://ec.europa.eu/research/swafs/gendered-innovations/index_en.cfm?pg=home. Accessed 11 Jan 2017.

  91. Day S, Mason R, Lagosky S, Rochon PA. Integrating and evaluating sex and gender in health research. Health Res Policy Syst. 2016;14:75. doi:10.1186/s12961-016-0147-7.

    Article  PubMed  PubMed Central  Google Scholar 

  92. Doull M, Runnels VE, Tudiver S, Boscoe M. Appraising the evidence: applying sex- and gender-based analysis (SGBA) to Cochrane systematic reviews on cardiovascular diseases. J Womens Health (Larchmt). 2010;19(5):997–1003. doi:10.1089/jwh.2009.1626.

    Article  Google Scholar 

  93. European Commission. DG for Research and Innovation. Toolkit Gender in EU-funded Research. 2011. http://bookshop.europa.eu/en/toolkit-gender-in-eufunded-research-pbKINA24840/. Accessed 11 Jan 2017.

  94. Jahn I, Frick J, Gansefort D. Assessing sex/gender sensitivity of epidemiological publications: development of a screening tool. Eur J Epidemiol. 2013;28:39.

    Google Scholar 

  95. Pederson A, Greaves L, Poole N. Gender-transformative health promotion for women: a framework for action. Health Promot Int. 2015;30:140–50. doi:10.1093/heapro/dau083.

    Article  PubMed  Google Scholar 

  96. Schiebinger L, Leopold SS, Miller VM. Editorial policies for sex and gender analysis. Lancet. 2016;388:2841–2. doi:10.1016/s0140-6736(16)32392-3.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We thank the scientific board of the project ‘Epi goes Gender’ and our colleagues Johann Frick, Dirk Gansefort, Hannah Jilani, Ina Schaefer, as well as Florence Samkange-Zeeb, Ronja Foraita, Iris Pigeot, and Hajo Zeeb for valuable comments on earlier versions of this paper. Additionally, we thank Gabriele Bolte, Enno Swart and Hajo Zeeb for their comments on the assessment instrument.

Funding

This research was conducted as part of the network “Gender-sensitive Research in Epidemiology, Neurosciences and Genetics/Tumour Research” [Sub-Project: Epi goes Gender]. The authors gratefully acknowledge the financial support of the German Federal Ministry of Education and Research (BMBF) (grant no. 01FP1058/59) and of the European Social Fund of the European Union (ESF). The responsibility for the content of this publication lies with the authors. The publication of this article was funded by the Open Access Fund of the Leibniz Association.

Availability of data and materials

All relevant data are within the text and additional files. For further information please contact the authors.

Authors’ contributions

IJ planned the study with contributions of CB and FG. IJ, CB and FG developed the assessment instrument and evaluated the articles. IJ and TB drafted the manuscript with contributions from CB and FG. All authors critically reviewed the drafts and approved the final version of this manuscript for submission.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ingeborg Jahn.

Additional file

Additional file 1:

Assessment instrument: Identification of good practice examples of sex/gender sensitive health research. (PDF 48 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jahn, I., Börnhorst, C., Günther, F. et al. Examples of sex/gender sensitivity in epidemiological research: results of an evaluation of original articles published in JECH 2006–2014. Health Res Policy Sys 15, 11 (2017). https://doi.org/10.1186/s12961-017-0174-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-017-0174-z

Keywords