In Defense of Unrepresentative Samples
kind of racist of you to assume stuff won't generalize to minorities
When I’m not cranking out Substack posts at full speed (/s) I mostly make surveys. Finding people to take these surveys is hard, so the two most popular methods academic researchers use are:
Recruiting undergraduates to complete your survey in exchange for course/extra credit in some of their classes.
Recruiting online samples using services like Prolific Academic or Amazon MTurk, and paying the participants who complete your survey on that site a small bonus (usually around $9/hr) for their time.
When people I meet learn that this is how I typically get the data for my research, they often react with a lot of skepticism. Aren’t those samples skewed heavily white, wealthy, educated, western, etc.? How can you claim to be studying human cognition with such a biased sample?
I think a lot of people took an intro to psych course in their undergrad that taught them that reacting this way is the right thing to do. It’s worth clarifying that often these recruitment methods can be made more representative before or after data collection (e.g., Prolific allows you to recruit participants who are nationally representative in terms of race, gender, and politics for an additional fee, and people typically do this for empirical surveys - there are also post-hoc stats tricks data analysts can use to make samples more balanced as a robustness check in many cases). But even in absence of these representativeness adjustments, I don’t really think it’s a big deal whether most social science research contains an unbiased sample.
My main reason for not caring about this too much is that the fundamental features of human cognition seem pretty consistent across different groups and cultures. For instance, this study aimed to replicate 13 classic social psych findings and found that although about a quarter of them didn’t replicate, the difference in effect sizes when the findings were tested in university labs vs. with representative U.S. samples vs. with representative international samples was consistently negligible. In other words, bad scientific methods, not unrepresentative samples, are responsible for the replication crisis. Even the vaguely psuedosciencey personality psych stuff seems to generalize surprisingly well across cultures.
Given that many patterns in human cognition appear to be nearly universal, it’s not clear to me why researchers should prioritize recruiting representative samples at significant cost to their time or research budgets. While there are certainly some cases where context matters a lot for interpreting a research finding (e.g., if researchers tried to make claims about the broader American political process using only findings from undergraduate samples, this would be dumb for obvious reasons), it seems pretty reasonable to treat most findings about even a narrow group of people’s thoughts and behaviors as substantial evidence about how other groups of people will think and behave, too, unless you can tell a compelling story to the contrary. It also feels vaguely racist when people insist otherwise. Like, why exactly are you so confident my study on probability estimation wouldn’t replicate in a sample with fewer white people…?
I think a better norm would be for people to interpret social science findings as probably broadly generalizable unless reasons suggest otherwise. Maybe it’s true that some cultures have different relationships with fundamental concepts in human decision-making like risk or communication! But the burden’s on you to make that case :)