Week 4: Science and Publication Bias

Instructions

Some studies have been conducted, but they have never been published. This week, we discuss the causes and consequences of publication bias and what it means for evidence-informed public policy making. Can systematic reviews, evidence maps, and/or meta-analysis help to navigate a biased body of evidence? What are their limits?

Required readings

• West, Jevin D and Bergstrom, Carl T (2021). Misinformation in and about Science. Proceedings of the National Academy of Sciences, 118(15), pp. e1912444117.

Also take a look at two “What works?” websites: Homelessness Effectiveness MapCrime Solutions. If you are curious, the Resource page lists more.

Further reading

• Frey, Bruno S (2003). Publishing as Prostitution?–Choosing Between One's Own Ideas and Academic Success. Public choice, 116(1), pp. 205–223.
• Pielke Jr, Roger A (2007). The Honest Broker: Making Sense of Science in Policy and Politics. Cambridge: Cambridge University Press.
• Franco, Annie and Malhotra, Neil and Simonovits, Gabor (2014). Publication Bias in the Social Sciences: Unlocking the File Drawer. Science, 345(6203), pp. 1502–1505.
• Ioannidis, John P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), pp. e124.

Another meta-analysis on nudging that received some critical attention in summer 2022:

• Mertens, Stephanie and Herberz, Mario and Hahnel, Ulf JJ and Brosch, Tobias (2022). The Effectiveness of Nudging: A Meta-analysis of Choice Architecture Interventions across Behavioral Domains. Proceedings of the National Academy of Sciences, 119(1), pp. e2107346118.
• Bakdash, Jonathan Z and Marusich, Laura R (2022). Left-truncated Effects and Overestimated Meta-analytic Means. Proceedings of the National Academy of Sciences, 119(31), pp. e2203616119.
• Maier, Maximilian and Bartoš, František and Stanley, TD and Shanks, David R and Harris, Adam JL and Wagenmakers, Eric-Jan (2022). No Evidence for Nudging After Adjusting for Publication Bias. Proceedings of the National Academy of Sciences, 119(31), pp. e2200300119.
• Szaszi, Barnabas and Higney, Anthony and Charlton, Aaron and Gelman, Andrew and Ziano, Ignazio and Aczel, Balazs and Goldstein, Daniel G and Yeager, David S and Tipton, Elizabeth (2022). No Reason to Expect Large and Consistent Effects of Nudge Interventions. Proceedings of the National Academy of Sciences, 119(31), pp. e2200732119.
• Mertens, Stephanie and Herberz, Mario and Hahnel, Ulf JJ and Brosch, Tobias (2022). Reply to Maier et al., Szaszi et al., and Bakdash and Marusich: The Present and Future of Choice Architecture Research. Proceedings of the National Academy of Sciences, 119(31), pp. e2202928119.

Suggested media

• Congratulations. Your Study Went Nowhere, New York Times, September 24, 2018.
• The Scientist, BBC Radio 4, October 24, 2002.