Eva Vivalt

I have been highly supportive of collecting ex ante forecasts of research results for some time now. Today, I am happy to say that the Social Science Prediction Platform is finally ready for public consumption.

This project, joint with Stefano DellaVigna and with the essential assistance of Nicholas Otis, Henry Xu, and the BITSS team, aims to do several things. Personally, I hope it can:

1. Popularize the ex ante prediction of research results

As argued elsewhere (e.g. here, here and here, as well as in this Science piece with Stefano DellaVigna and Devin Pope), ex ante priors are extremely useful scientifically. Some people already routinely collect them, but to others this is still a relatively new idea.

Personally, I hope that the collection of forecasts of research results becomes somewhat like pre-analysis plans: widespread in certain fields. However, like pre-analysis plans, it may take time for collecting forecasts to really take off. First, we need good examples, templates, and even workshops. This SSPP can be a useful hub to organize these activities. We already have an extensive Forecasting Survey Guide with tips for how to structure forecasting surveys, based on earlier work. We have a template and some annotated examples. We’ve had a workshop on the topic and hope to have more in the future. It is my hope that by gathering these resources together, we can provide a public good that enables others to launch research projects on a variety of topics, such as on meta-science, forecasting, or belief updating, in addition to increasing the value of each individual forecasted study.

2. Solve the coordination problem inherent in gathering forecasts

What coordination problem? Well, if everyone gathered forecasts for all their projects, their private incentive would be to request as many forecasts as possible to maximize their sample size. However, the more requests you receive as a forecaster, the more likely you are to ignore those requests. The platform can help to solve this problem in several ways:

– It can nudge researchers towards sending their surveys to fewer individuals. Pilots suggest that often many forecasts are not needed for a precise estimate

– The platform knows when an individual has already taken a lot of surveys and can direct further surveys to other people instead. For example, someone can specify: I only want to forecast the results of 5 surveys per month. Then we don’t send them more than that. I only wish journals had a similar system whereby they could coordinate across journals and not send anyone too many requests to referee

– We can build in incentives and nudges for people to provide more forecasts. For example, we are planning an incentive scheme for graduate student forecasters, and we suggest that those who want to have forecasts collected for their own projects provide a certain number of forecasts themselves, to be fair.

3. Gather panel data and identify super-forecasters

An advantage that the platform has over individual research teams gathering forecasts for their own independent studies is that we can begin to look across predictions and follow forecasters longitudinally. This could be immensely valuable in identifying super-forecasters and learning more about which types of studies or results are easier to forecast and to what extent forecast ability depends on various forecaster characteristics such as domain expertise.

The first few studies are now up online and ready to be forecast! Over the coming weeks we will post other studies and slowly open it up, so please do get in touch if you have a survey you would like to run. We expect that we’ll continue to learn as we go and would welcome further feedback.

One of the more important things I’ve changed my mind about recently is the best cause to donate to. I now put the most credence on the possibility that the best option is donating to a fund that invests the money and disburses strategically in the future. I will refer to this as “giving later”, though I actually support giving now to a donor-advised fund set up to disburse in the future, for the value that donating now can have for encouraging others to donate (and because of the risk that even if one thinks one will donate later, one will at some point change one’s mind).

There are several reasons why I prefer a fund that disburses in the future. First, I believe people currently discount the future too much (see hyperbolic discounting, climate change). If people discount the future, that causes the rate of return on investments to always be higher than the growth rate (else people would not be willing to invest). In economics, the Ramsey equation is often used to determine how much a social planner should discount future consumption. It is specified by r=ηg+δ, where r is the real rate of return on investment, η is the extent to which marginal utility decreases with consumption, g is the growth rate, and δ represents pure time preferences. Unless one personally puts a particularly high value on δ, it makes sense to invest today and spend later to take advantage of the gap between the real rate of return on investment (~7%) and the growth rate (~3-4%).

How should one set δ? This is a huge open question. Like most effective altruists, I do not believe one should treat people today any differently from people tomorrow. But one might still wish to place a non-zero value on δ due to the risk that people will simply not exist in the future – that nuclear war or other disasters will wipe them out. Economists tend to like to respect people’s pure time preferences and so end up with rather higher values than effective altruists. The Stern report famously set δ=0.1, while Nordhaus prefers δ=3. The current Trump administration set δ up to 7, which justifies not doing anything about climate change (see also this nice figure). With a modest δ, it makes sense to invest now and give later according to the Ramsey equation.

A second reason that I prefer a fund that disburses in the future is that I think we have very limited knowledge today and that our knowledge is increasing. I am concerned about the problem that research results do not generalize all that well, but with respect to economic development I am optimistic that the situation can improve. With respect to technological change which could bring huge benefits or risks, I think we know even less about the problems future generations will face and may be able to understand them better in the future. It seems unlikely to me that we are at the exact moment in time, out of all periods of time from here on out into the future, that we actually have the best opportunity to do good. We may not recognize the best moment when it comes, but that just pushes the argument back a step: I also think it unlikely that we are at the best moment, out of the whole foreseeable future, to have the best combination of knowledge and opportunity to do good.

These are not novel arguments. Some form of them is made in several other blog posts, for example. Some of the criticisms commonly raised are whether donations today can help to improve the long-run growth rate and whether it is feasible to design and maintain a fund that disburses later without value drift. There are sadly few long-run follow-ups of development interventions, but it seems prima facie unlikely that interventions will have a long-run effect on the growth rate, given the growth rate is a function of many, many things. I expect most effects to taper off over time but acknowledge that further research in this area is needed. With regards to it being difficult to build a persistent and safe institution, I agree that this is challenging, but not altogether impossible, and I know several people working on this right now.

There are several reasons to be optimistic. First, this institution could take into consideration the risk of e.g. nuclear war or values drift in setting its disbursement scheme, so that it has a more aggressive disbursement scheme as the risks go up (in the extreme case, disbursing everything right away). Second, it is easy to think of a “lower-bound” version of this that would not be at much risk for values drift. For example, suppose a fund existed that disbursed the minimum amount possible every year (U.S. charities, for example, are required to disburse 5% per year), and then disbursed the rest in year 10. In the simplest possible version of this, think of a cash transfer charity like GiveDirectly which gives out cash to people in developing countries via mobile money transfers. One could set up the institution to automatically make these payments over time without any deviations allowed (say, through a smart contract). Unless mobile money is no longer in use 10 years from now, this option would seem to strictly dominate giving cash transfers today. What about other types of transfers, like to some of GiveWell’s top-rated charities, the Against Malaria Foundation or Deworm the World? It is possible that interventions are particularly cheap now, while they may be more expensive (for the same benefit) in the future. For example, most of the gains in life expectancy have been due to improvements in sanitation and basic healthcare reducing under-5 mortality; it is a lot harder to increase life expectancy from 79 to 80. There are some arguments that can be made against this. I won’t get into them too much, though I will note that under some conditions this situation could be addressed by letting the investments compound for longer before using them. In any case, my assumption is that if the calculus really works out this way, we are back in the world in which the organization disburses everything right away. Further, if one considers the farther future and cares about future potential lives, one may wish to place more emphasis on avoiding existential or extinction risks, and it is not clear that we are at a particularly good time in history to do that.

I think it appeals psychologically to many people – myself included – to think that we are living at a particularly important time. However, I recognize that people have thought this throughout history. As more time has passed, I have become increasingly confident that my gut antipathy to the idea that it’s better to “give later” is just a cognitive bias.

For those wanting to attend the Forecasts of Social Science Results workshop on Dec. 11 remotely, please use this link to join via Zoom. (If you haven’t used Zoom before, you will be asked to install software on your computer or mobile device before joining.)

An updated agenda is posted here.