Eva Vivalt

The Social Science Prediction Platform is hiring for a pre-doctoral fellow and a full-stack developer. Please see the job adverts below.

---

Pre-Doctoral Fellowship (filled)

Job description: The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece. 

Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are seeking a pre-doctoral research associate to assist with the Social Science Prediction Platform and be an integral part of the team. The position will be full-time and last for a minimum of one year with the possibility of extension. The Fellow will be based in the Department of Economics at the University of Toronto. Work authorization in Canada, whether by citizenship or an open work permit, is strictly required. Past research assistants working with Eva Vivalt on other projects have subsequently gained admission to top PhD programs in Economics and Political Science.

Duties may include:

• Designing and testing forecasting surveys for randomized controlled trials in a number of fields, including behavioral economics, development economics, experimental economics, and labor economics;
• Identifying candidate projects that would particularly benefit from the collection of forecasts;
• Liaising with researchers at other universities regarding their projects;
• Cleaning data related to the project;
• Participating in weekly team meetings with the PIs and other staff.

Minimum required education and experience:

• Bachelor’s degree in economics, computer science, mathematics, statistics, or a related field;
• Excellent communication skills in English.

Preferred education and experience:

• Previous research experience, such as through past research assistantships or an independent research project;
• Strong quantitative background;
• Experience with Qualtrics survey software;
• Experience with a programming language such as Stata, R, or Python;
• An interest in pursuing a PhD in Economics or a related field.

Start date: as soon as possible

Location: Toronto, Canada

How to apply: please e-mail a CV to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “Pre-doctoral fellowship”.

Deadline: August 9, 2022. Applications submitted after the deadline will be considered on a rolling basis.

---

Full-Stack Developer

Job description: Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are looking to hire a senior software developer with web development experience to take the lead in updating and expanding the Social Science Prediction Platform (SSPP) at scale. The SSPP facilitates the collection and cataloging of forecasts of research results. It uses the Qualtrics API to integrate surveys programmed in Qualtrics on the SSPP interface and a relational database to connect survey responses to project objects and user data such as contact information, demographics, and preferences. The platform generates complex, large data sets which must be kept confidential.

The developer must have work authorization in Canada or the United States.

The SSPP requires significant extensions and updates to efficiently scale up, including:

• Improving the usability and navigability of the database structure for users with admin privileges;
• Improving navigability of the project catalog;
• Improving the visual presentation of survey results;
• Automated notification to respondents according to their preferences and past completion of forecasting surveys;
• Ensuring the Qualtrics and Sendgrid APIs continue to function well with evolving  platform features;
• Enabling users to edit Qualtrics .qsf files dynamically online;
• Implementing a secure single sign-on system in collaboration with external platforms;
• Developing a systemized approach to measuring the accuracy of forecasts and providing these accuracy measures to forecasters;
• Developing a system to automatically pay forecasters according to their performance and ensuring the security of this system;
• Identifying and merging duplicate accounts; 
• Building a dynamic “leaderboard” to publicly reward forecasters that provide accurate forecasts.

Required qualifications:

• Experience with Python/Django/JavaScript (we use Django/Jinja2 as our main framework);
• Experience with DevOps (we use Docker locally for development and deploy using Heroku, though we would also like the developer to be responsible for migrating the site to AWS);
• Ability to design and work with REST APIs (e.g., our platform calls Qualtrics APIs to work with survey data, and SendGrid APIs to enable communication with users and stakeholders);
• Experience with database management, specifically Postgres; 
• Excellent understanding of architectural frameworks and data warehousing to design solutions for large, complex data sets;
• Excellent SQL and data manipulation skills;
• Experience with SSO providers;
• Experience developing secure payment systems;
• Back-end development experience, including experience building the back-end of a database or website in which datasets are being queried, manipulated, and transferred to users;
• Front-end development experience (e.g., building out intuitive user interfaces, SSR, React, cross-origin communication);
• Excellent communication skills, including the ability to assess and communicate technical challenges and design decisions.

Preferred qualifications:

• Some understanding of empirical economic/data science methods;
• Commitment to producing clean, well-documented code (e.g., as evidenced by public GitHub repos).

Start date: as soon as possible

Location:

• Option to work remotely: meetings during U.S. Pacific Time hours
• Option to work in Berkeley, CA or Toronto, ON
• Must have legal ability to work in the United States or Canada 

Competitive salary.

How to apply: please e-mail a CV/resume and expression of interest to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “developer application”.

I have been highly supportive of collecting ex ante forecasts of research results for some time now. Today, I am happy to say that the Social Science Prediction Platform is finally ready for public consumption.

This project, joint with Stefano DellaVigna and with the essential assistance of Nicholas Otis, Henry Xu, and the BITSS team, aims to do several things. Personally, I hope it can:

1. Popularize the ex ante prediction of research results

As argued elsewhere (e.g. here, here and here, as well as in this Science piece with Stefano DellaVigna and Devin Pope), ex ante priors are extremely useful scientifically. Some people already routinely collect them, but to others this is still a relatively new idea.

Personally, I hope that the collection of forecasts of research results becomes somewhat like pre-analysis plans: widespread in certain fields. However, like pre-analysis plans, it may take time for collecting forecasts to really take off. First, we need good examples, templates, and even workshops. This SSPP can be a useful hub to organize these activities. We already have an extensive Forecasting Survey Guide with tips for how to structure forecasting surveys, based on earlier work. We have a template and some annotated examples. We’ve had a workshop on the topic and hope to have more in the future. It is my hope that by gathering these resources together, we can provide a public good that enables others to launch research projects on a variety of topics, such as on meta-science, forecasting, or belief updating, in addition to increasing the value of each individual forecasted study.

2. Solve the coordination problem inherent in gathering forecasts

What coordination problem? Well, if everyone gathered forecasts for all their projects, their private incentive would be to request as many forecasts as possible to maximize their sample size. However, the more requests you receive as a forecaster, the more likely you are to ignore those requests. The platform can help to solve this problem in several ways:

– It can nudge researchers towards sending their surveys to fewer individuals. Pilots suggest that often many forecasts are not needed for a precise estimate

– The platform knows when an individual has already taken a lot of surveys and can direct further surveys to other people instead. For example, someone can specify: I only want to forecast the results of 5 surveys per month. Then we don’t send them more than that. I only wish journals had a similar system whereby they could coordinate across journals and not send anyone too many requests to referee

– We can build in incentives and nudges for people to provide more forecasts. For example, we are planning an incentive scheme for graduate student forecasters, and we suggest that those who want to have forecasts collected for their own projects provide a certain number of forecasts themselves, to be fair.

3. Gather panel data and identify super-forecasters

An advantage that the platform has over individual research teams gathering forecasts for their own independent studies is that we can begin to look across predictions and follow forecasters longitudinally. This could be immensely valuable in identifying super-forecasters and learning more about which types of studies or results are easier to forecast and to what extent forecast ability depends on various forecaster characteristics such as domain expertise.

The first few studies are now up online and ready to be forecast! Over the coming weeks we will post other studies and slowly open it up, so please do get in touch if you have a survey you would like to run. We expect that we’ll continue to learn as we go and would welcome further feedback.

One of the more important things I’ve changed my mind about recently is the best cause to donate to. I now put the most credence on the possibility that the best option is donating to a fund that invests the money and disburses strategically in the future. I will refer to this as “giving later”, though I actually support giving now to a donor-advised fund set up to disburse in the future, for the value that donating now can have for encouraging others to donate (and because of the risk that even if one thinks one will donate later, one will at some point change one’s mind).

There are several reasons why I prefer a fund that disburses in the future. First, I believe people currently discount the future too much (see hyperbolic discounting, climate change). If people discount the future, that causes the rate of return on investments to always be higher than the growth rate (else people would not be willing to invest). In economics, the Ramsey equation is often used to determine how much a social planner should discount future consumption. It is specified by r=ηg+δ, where r is the real rate of return on investment, η is the extent to which marginal utility decreases with consumption, g is the growth rate, and δ represents pure time preferences. Unless one personally puts a particularly high value on δ, it makes sense to invest today and spend later to take advantage of the gap between the real rate of return on investment (~7%) and the growth rate (~3-4%).

How should one set δ? This is a huge open question. Like most effective altruists, I do not believe one should treat people today any differently from people tomorrow. But one might still wish to place a non-zero value on δ due to the risk that people will simply not exist in the future – that nuclear war or other disasters will wipe them out. Economists tend to like to respect people’s pure time preferences and so end up with rather higher values than effective altruists. The Stern report famously set δ=0.1, while Nordhaus prefers δ=3. The current Trump administration set δ up to 7, which justifies not doing anything about climate change (see also this nice figure). With a modest δ, it makes sense to invest now and give later according to the Ramsey equation.

A second reason that I prefer a fund that disburses in the future is that I think we have very limited knowledge today and that our knowledge is increasing. I am concerned about the problem that research results do not generalize all that well, but with respect to economic development I am optimistic that the situation can improve. With respect to technological change which could bring huge benefits or risks, I think we know even less about the problems future generations will face and may be able to understand them better in the future. It seems unlikely to me that we are at the exact moment in time, out of all periods of time from here on out into the future, that we actually have the best opportunity to do good. We may not recognize the best moment when it comes, but that just pushes the argument back a step: I also think it unlikely that we are at the best moment, out of the whole foreseeable future, to have the best combination of knowledge and opportunity to do good.

These are not novel arguments. Some form of them is made in several other blog posts, for example. Some of the criticisms commonly raised are whether donations today can help to improve the long-run growth rate and whether it is feasible to design and maintain a fund that disburses later without value drift. There are sadly few long-run follow-ups of development interventions, but it seems prima facie unlikely that interventions will have a long-run effect on the growth rate, given the growth rate is a function of many, many things. I expect most effects to taper off over time but acknowledge that further research in this area is needed. With regards to it being difficult to build a persistent and safe institution, I agree that this is challenging, but not altogether impossible, and I know several people working on this right now.

There are several reasons to be optimistic. First, this institution could take into consideration the risk of e.g. nuclear war or values drift in setting its disbursement scheme, so that it has a more aggressive disbursement scheme as the risks go up (in the extreme case, disbursing everything right away). Second, it is easy to think of a “lower-bound” version of this that would not be at much risk for values drift. For example, suppose a fund existed that disbursed the minimum amount possible every year (U.S. charities, for example, are required to disburse 5% per year), and then disbursed the rest in year 10. In the simplest possible version of this, think of a cash transfer charity like GiveDirectly which gives out cash to people in developing countries via mobile money transfers. One could set up the institution to automatically make these payments over time without any deviations allowed (say, through a smart contract). Unless mobile money is no longer in use 10 years from now, this option would seem to strictly dominate giving cash transfers today. What about other types of transfers, like to some of GiveWell’s top-rated charities, the Against Malaria Foundation or Deworm the World? It is possible that interventions are particularly cheap now, while they may be more expensive (for the same benefit) in the future. For example, most of the gains in life expectancy have been due to improvements in sanitation and basic healthcare reducing under-5 mortality; it is a lot harder to increase life expectancy from 79 to 80. There are some arguments that can be made against this. I won’t get into them too much, though I will note that under some conditions this situation could be addressed by letting the investments compound for longer before using them. In any case, my assumption is that if the calculus really works out this way, we are back in the world in which the organization disburses everything right away. Further, if one considers the farther future and cares about future potential lives, one may wish to place more emphasis on avoiding existential or extinction risks, and it is not clear that we are at a particularly good time in history to do that.

I think it appeals psychologically to many people – myself included – to think that we are living at a particularly important time. However, I recognize that people have thought this throughout history. As more time has passed, I have become increasingly confident that my gut antipathy to the idea that it’s better to “give later” is just a cognitive bias.