Eva Vivalt

Location: Toronto, Canada

Job description: The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece. 

Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are seeking a pre-doctoral research associate to assist with the Social Science Prediction Platform and be an integral part of the team. The position will be full-time and last for a minimum of one year with the possibility of extension. The Fellow will be based in the Department of Economics at the University of Toronto. Work authorization in Canada, whether by citizenship or an open work permit, is strictly required. Past research assistants working with Eva Vivalt on other projects have subsequently gained admission to top PhD programs in Economics and Political Science.

Duties may include:

• Cleaning and analyzing data related to the project;
• Designing and testing forecasting surveys for randomized controlled trials in a number of fields, including behavioral economics, development economics, experimental economics, and labor economics;
• Liaising with researchers at other universities regarding their projects;
• Participating in weekly team meetings with the PIs and other staff.

Minimum required education and experience:

• Bachelor’s degree in economics, computer science, mathematics, statistics, or a related field.

Preferred education and experience:

• Experience with a programming language such as Stata, R, or Python;
• Previous research experience, such as through past research assistantships or an independent research project;
• Strong quantitative background;
• Experience with Qualtrics survey software;
• An interest in pursuing a PhD in Economics or a related field.

Start date: as soon as possible

How to apply: please fill out an application here to apply.

The Social Science Prediction Platform is hiring for a pre-doctoral fellow and a full-stack developer. Please see the job adverts below.

---

Pre-Doctoral Fellowship (filled)

Job description: The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece. 

Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are seeking a pre-doctoral research associate to assist with the Social Science Prediction Platform and be an integral part of the team. The position will be full-time and last for a minimum of one year with the possibility of extension. The Fellow will be based in the Department of Economics at the University of Toronto. Work authorization in Canada, whether by citizenship or an open work permit, is strictly required. Past research assistants working with Eva Vivalt on other projects have subsequently gained admission to top PhD programs in Economics and Political Science.

Duties may include:

• Designing and testing forecasting surveys for randomized controlled trials in a number of fields, including behavioral economics, development economics, experimental economics, and labor economics;
• Identifying candidate projects that would particularly benefit from the collection of forecasts;
• Liaising with researchers at other universities regarding their projects;
• Cleaning data related to the project;
• Participating in weekly team meetings with the PIs and other staff.

Minimum required education and experience:

• Bachelor’s degree in economics, computer science, mathematics, statistics, or a related field;
• Excellent communication skills in English.

Preferred education and experience:

• Previous research experience, such as through past research assistantships or an independent research project;
• Strong quantitative background;
• Experience with Qualtrics survey software;
• Experience with a programming language such as Stata, R, or Python;
• An interest in pursuing a PhD in Economics or a related field.

Start date: as soon as possible

Location: Toronto, Canada

How to apply: please e-mail a CV to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “Pre-doctoral fellowship”.

Deadline: August 9, 2022. Applications submitted after the deadline will be considered on a rolling basis.

---

Full-Stack Developer

Job description: Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are looking to hire a senior software developer with web development experience to take the lead in updating and expanding the Social Science Prediction Platform (SSPP) at scale. The SSPP facilitates the collection and cataloging of forecasts of research results. It uses the Qualtrics API to integrate surveys programmed in Qualtrics on the SSPP interface and a relational database to connect survey responses to project objects and user data such as contact information, demographics, and preferences. The platform generates complex, large data sets which must be kept confidential.

The developer must have work authorization in Canada or the United States.

The SSPP requires significant extensions and updates to efficiently scale up, including:

• Improving the usability and navigability of the database structure for users with admin privileges;
• Improving navigability of the project catalog;
• Improving the visual presentation of survey results;
• Automated notification to respondents according to their preferences and past completion of forecasting surveys;
• Ensuring the Qualtrics and Sendgrid APIs continue to function well with evolving  platform features;
• Enabling users to edit Qualtrics .qsf files dynamically online;
• Implementing a secure single sign-on system in collaboration with external platforms;
• Developing a systemized approach to measuring the accuracy of forecasts and providing these accuracy measures to forecasters;
• Developing a system to automatically pay forecasters according to their performance and ensuring the security of this system;
• Identifying and merging duplicate accounts; 
• Building a dynamic “leaderboard” to publicly reward forecasters that provide accurate forecasts.

Required qualifications:

• Experience with Python/Django/JavaScript (we use Django/Jinja2 as our main framework);
• Experience with DevOps (we use Docker locally for development and deploy using Heroku, though we would also like the developer to be responsible for migrating the site to AWS);
• Ability to design and work with REST APIs (e.g., our platform calls Qualtrics APIs to work with survey data, and SendGrid APIs to enable communication with users and stakeholders);
• Experience with database management, specifically Postgres; 
• Excellent understanding of architectural frameworks and data warehousing to design solutions for large, complex data sets;
• Excellent SQL and data manipulation skills;
• Experience with SSO providers;
• Experience developing secure payment systems;
• Back-end development experience, including experience building the back-end of a database or website in which datasets are being queried, manipulated, and transferred to users;
• Front-end development experience (e.g., building out intuitive user interfaces, SSR, React, cross-origin communication);
• Excellent communication skills, including the ability to assess and communicate technical challenges and design decisions.

Preferred qualifications:

• Some understanding of empirical economic/data science methods;
• Commitment to producing clean, well-documented code (e.g., as evidenced by public GitHub repos).

Start date: as soon as possible

Location:

• Option to work remotely: meetings during U.S. Pacific Time hours
• Option to work in Berkeley, CA or Toronto, ON
• Must have legal ability to work in the United States or Canada 

Competitive salary.

How to apply: please e-mail a CV/resume and expression of interest to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “developer application”.

I have been highly supportive of collecting ex ante forecasts of research results for some time now. Today, I am happy to say that the Social Science Prediction Platform is finally ready for public consumption.

This project, joint with Stefano DellaVigna and with the essential assistance of Nicholas Otis, Henry Xu, and the BITSS team, aims to do several things. Personally, I hope it can:

1. Popularize the ex ante prediction of research results

As argued elsewhere (e.g. here, here and here, as well as in this Science piece with Stefano DellaVigna and Devin Pope), ex ante priors are extremely useful scientifically. Some people already routinely collect them, but to others this is still a relatively new idea.

Personally, I hope that the collection of forecasts of research results becomes somewhat like pre-analysis plans: widespread in certain fields. However, like pre-analysis plans, it may take time for collecting forecasts to really take off. First, we need good examples, templates, and even workshops. This SSPP can be a useful hub to organize these activities. We already have an extensive Forecasting Survey Guide with tips for how to structure forecasting surveys, based on earlier work. We have a template and some annotated examples. We’ve had a workshop on the topic and hope to have more in the future. It is my hope that by gathering these resources together, we can provide a public good that enables others to launch research projects on a variety of topics, such as on meta-science, forecasting, or belief updating, in addition to increasing the value of each individual forecasted study.

2. Solve the coordination problem inherent in gathering forecasts

What coordination problem? Well, if everyone gathered forecasts for all their projects, their private incentive would be to request as many forecasts as possible to maximize their sample size. However, the more requests you receive as a forecaster, the more likely you are to ignore those requests. The platform can help to solve this problem in several ways:

– It can nudge researchers towards sending their surveys to fewer individuals. Pilots suggest that often many forecasts are not needed for a precise estimate

– The platform knows when an individual has already taken a lot of surveys and can direct further surveys to other people instead. For example, someone can specify: I only want to forecast the results of 5 surveys per month. Then we don’t send them more than that. I only wish journals had a similar system whereby they could coordinate across journals and not send anyone too many requests to referee

– We can build in incentives and nudges for people to provide more forecasts. For example, we are planning an incentive scheme for graduate student forecasters, and we suggest that those who want to have forecasts collected for their own projects provide a certain number of forecasts themselves, to be fair.

3. Gather panel data and identify super-forecasters

An advantage that the platform has over individual research teams gathering forecasts for their own independent studies is that we can begin to look across predictions and follow forecasters longitudinally. This could be immensely valuable in identifying super-forecasters and learning more about which types of studies or results are easier to forecast and to what extent forecast ability depends on various forecaster characteristics such as domain expertise.

The first few studies are now up online and ready to be forecast! Over the coming weeks we will post other studies and slowly open it up, so please do get in touch if you have a survey you would like to run. We expect that we’ll continue to learn as we go and would welcome further feedback.