NTT Corporation (Headquarters: Chiyoda Ward, Tokyo; Representative Member of the Board and President: Akira Shimada) and the Cybersecurity Laboratory of the National Institute of Information and Communications Technology (NICT, President: Hideyuki Tokuda, Ph.D.) have conducted a systematic review of user studies in the security and privacy field. The analysis quantitatively highlights that many existing studies focus predominantly on populations from limited geographical and cultural regions, mostly in Western countries. This geographical skew points to a significant limitation in the generalizability of previous security and privacy research, suggesting that individuals from other regions, such as Japan and other parts of Asia, may not fully benefit from these findings. The study underscores the need to recognize and address these regional and cultural differences and proposes research methodologies to better understand and incorporate diverse populations. This research was presented at USENIX Security 2024, a leading international cybersecurity conference, held in Philadelphia, USA, from August 14-16, 2024.1. Background of the Study
In research areas focused on human subjects, such as psychology and human-computer interaction (HCI), user studies have been instrumental in revealing psychological and behavioral traits. However, these studies have faced criticism for their Western-centric bias, predominantly focusing on “WEIRD” populations—those from Western, Educated, Industrialized, Rich, and Democratic societies. Previous research has not thoroughly assessed whether the findings from these geographically skewed studies are universally applicable or if significant regional differences exist.
In the field of security and privacy research—where studies analyze psychological, behavioral, and decision-making processes to inform the design, implementation, and operation of computer systems—some research has noted the impact of geographical and cultural differences. However, the extent of the Western bias in security and privacy research remains unclear.
2. Outline of the Study
Using a systematic literature review approach, we examined user studies published in security and privacy research papers. We reviewed 7,587 papers presented at major international conferences on cybersecurity and human-computer interaction from 2017 to 2021. From this corpus, we identified 715 papers that conducted user studies in the security and privacy domain. We analyzed these papers for details on participants’ countries of residence, demographics, recruitment methods, study methodologies, and research topics, ensuring inter-rater reliability through multiple analysts.
3. Research Findings
Our analysis revealed that, in the security and privacy field, the proportion of user study samples from non-Western populations declined from 25% to 20% over the past five years (2017-2021), indicating an increased Western bias.
In contrast, a similar study in the HCI field showed an increase in non-Western samples from 16% to 30% over a five-year period (2016-2020), reflecting a trend towards reducing Western-centric bias. This comparison underscores that the Western skew in security and privacy research is more pronounced than in HCI researchOne significant factor contributing to the Western-centric bias in user studies is the geographical distribution of the researchers themselves. Our analysis revealed that 86.5% of the papers examined were authored exclusively by researchers from Western institutions. This geographical concentration leads researchers to recruit participants who are more readily accessible due to geographical and linguistic barriers, resulting in convenience sampling. This practice exacerbates the bias toward “WEIRD” populations in user studies.
To address this skew and improve our understanding of diverse populations, we propose the following approaches:
- Promotion of Replication Studies:
- Conduct replication studies involving non-WEIRD populations to enhance the generalizability of findings and identify differences across various geographical and cultural contexts.
- Overcoming Geographical and Linguistic Barriers:
- Utilize local crowdsourcing platforms commonly used in the regions where study participants are located.
- Increase researcher diversity by collaborating with local researchers who are familiar with the language, culture, and environment of the target population.
Outlook:
These strategies aim to foster international collaboration and enhance research diversity in the security and privacy field. By developing technologies that address the needs of a broader range of populations, we hope to advance inclusivity and contribute to the creation of more universally applicable security and privacy solutions..
