Several of my research projects have involved collecting priors from policymakers, practitioners and researchers (e.g. this and this). I think that collecting priors is quite important and undervalued in economics.
They have several uses:
1) They can help you prioritize outcomes or tweak other features of your design
If you know that there is more disagreement as to whether an intervention will affect a certain set of outcomes, you can focus your attention on that set of outcomes. This can help maximize learning and hopefully ensure your work is widely cited.
2) They help you avoid the problem that, regardless of what results you find, people say they knew it already
Have you ever done a study and then had people say they knew the results already, when you’re pretty sure they didn’t? It would be really nice to avoid this situation and keep your research from being overly discounted.
3) They enable learning about updating
If you collect priors, you can also collect posteriors and start to say something about how people interpret evidence and what behavioural biases a group of people might have, as in my paper with Aidan Coville on how policymakers, practitioners and researchers update.
4) They can make null results more interesting
Researchers currently aren’t given much credit for null results, a problem that can lead to specification searching. However, if we know a priori that some null results were completely unexpected, they become more interesting and informative.
For all these reasons, I am happy to say that due to a SSMART-funded project, which gathered priors from researchers and policymakers on their priors regarding the size of various interventions’ impacts, the World Bank’s Development Impact Evaluation group (DIME) is now capturing priors across their portfolio of impact evaluations through their monitoring system. This should lead to a large corpus of priors that can be very helpful in the future.
What do you think? Have you heard of any other interesting work eliciting priors?