:::: MENU ::::

An update and personal reflections about AidGrade

I have some very good news to share. As many of you know, back in 2012 I set up AidGrade, a small non-profit research institute, to collect the results of impact evaluations and synthesize them.

Fast-forward 11 years. A global consortium of institutions, led by the World Bank, is going to be working on an open repository of impact evaluation results that could be used for meta-analysis and policy (the Impact Data and Evidence Aggregation Library, or IDEAL). This is really close to AidGrade’s mission, and we will be participating in the consortium, helping to design the protocols, contribute data, and perform cross-checks with the other groups.

Apart from being generally excited about this soon-to-exist global public good, I am thrilled with the impact AidGrade has had. We made a case that this was a thing that should exist, and over time enough other people agreed that it will soon be a much larger thing (in which AidGrade will play the smallest of roles). All along, I was hoping that there could be a better institutional home for such a repository, and here we are. It’s the best possible outcome.

To anyone who supported AidGrade, either through time or money over the years, I hope you feel pleased with what you helped accomplish with AidGrade, and I hope you are as excited as I am about IDEAL.


Current plans as the incoming director of the Global Priorities Institute

I am taking leave from the University of Toronto to serve as the Director of the Global Priorities Institute (GPI) at the University of Oxford. I can’t express enough gratitude to the University of Toronto for enabling this. (I’ll be back in the fall to fulfill my teaching obligations, though – keep inviting me to seminars and such!)

GPI is an interdisciplinary research institute focusing on academic research that informs decision-makers on how to do good more effectively. In its first few years, under the leadership of its founding director, Hilary Greaves, GPI created and grew a community of academics in philosophy and economics interested in global priorities research. I am excited to build from this strong foundation and, in particular, to further develop the economics side.

There are several areas I would like to focus on while at GPI. The below items reflect my current views, however, I expect these views to be refined over time. These items are not intended to be an exhaustive list, but they are things I would like GPI to do more of on the margin.

1) Research on decision-making under uncertainty

There is a lot of uncertainty in estimates of the effects of various actions. My views here are coloured by my past work. In the early 2010s, I tried to compile estimates of the effects of popular development interventions such as insecticide-treated bed nets for malaria, deworming drugs, and unconditional cash transfers. My initial thought was that by synthesizing the evidence, I’d be able to say something more conclusive about “the best” intervention for a given outcome. Unfortunately, I found that results varied, a lot (you can read more about it in my JEEA paper).

If it’s really hard to predict effects in global development, which is a very well-studied area, it would seem even harder to know what to do in other areas with less evidence. Yet, decisions still have to be made. One of the core areas GPI has focused on in the past is decision-making under uncertainty, and I expect that to continue to be a priority research area. Some work on robustness might also fall under this category.

2) Increasing empirical research

GPI is an interdisciplinary institute combining philosophy and economics. To date, the economics side has largely focused on theoretical issues. But I think it’s important for there to be careful, rigorous empirical work at GPI. I think there are relevant hypotheses that can be tested that pertain to global priorities research.

Many economists interested in global priorities research come from applied fields like development economics, and there’s a talented pool of people who can do empirical work on, e.g., encouraging better uptake of evidence or forecasting. There’s simply a lot to be done here, and I look forward to working with colleagues like Julian Jamison (on leave from Exeter), Benjamin Tereick, and Mattie Toma (visiting from Warwick Business School), among many others.

3) Expanding GPI’s network in economics

There is an existing program at GPI for senior research affiliates based at other institutions. However, I think a lot more can be done with this, especially on the economics side. I’m still exploring the right structures, but suffice it to say, if you are an academic economist interested in global priorities research, please do get in touch. I am envisioning a network of loosely affiliated individuals in core fields of interest who would be sent notifications about research and funding opportunities. There may also be the occasional workshop or conference invitation.

4) Exploring expanding to other fields and topics

There are a number of topics that appear relevant to global priorities research that are not currently established at GPI. One field that we are trying to expand into is psychology. Within the existing economics and philosophy teams, we are also looking into whether there are any useful ways we can contribute to conversations around AI, as AI has the potential to be highly consequential in the near future.

5) Mentoring students and early career researchers

Young people are often interested in global priorities research, but an academic career can be difficult to navigate. GPI already has several programs targeted at students, such as the Global Priorities Fellowship Programme, the Open Student Workshop on Global Priorities Research, and a pre-doctoral fellowship program. However, academia can be notoriously competitive, so we will also pay more attention to mentoring early career researchers, including supporting researchers in finding external mentors where appropriate.

Finally, it is unfortunate for there to be a large pool of talented people that is relatively untapped. I think it is important to continue working on improving the diversity and reach of GPI. GPI has several initiatives to try to reach talented students, and I expect to see continued improvement here.

 

While I am directing GPI, I will also be continuing on with my own research, including work on the three largest US guaranteed income programs and improving the evidence-to-policy pipeline. That is all to say, it’s going to be a very busy few years, and I won’t be able to respond to all e-mails. But if you are interested in contributing in some way to global priorities research, please do get in touch! GPI is also planning to refresh the research agenda over the summer, so feedback is very welcome.


Research Assistant / Pre-Doc Position

Location: Toronto, Canada

Job description: The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece. 

Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are seeking a pre-doctoral research associate to assist with the Social Science Prediction Platform and be an integral part of the team. The position will be full-time and last for a minimum of one year with the possibility of extension. The Fellow will be based in the Department of Economics at the University of Toronto. Work authorization in Canada, whether by citizenship or an open work permit, is strictly required. Past research assistants working with Eva Vivalt on other projects have subsequently gained admission to top PhD programs in Economics and Political Science.

Duties may include:

  • Cleaning and analyzing data related to the project;
  • Designing and testing forecasting surveys for randomized controlled trials in a number of fields, including behavioral economics, development economics, experimental economics, and labor economics;
  • Liaising with researchers at other universities regarding their projects;
  • Participating in weekly team meetings with the PIs and other staff.

Minimum required education and experience:

  • Bachelor’s degree in economics, computer science, mathematics, statistics, or a related field.

Preferred education and experience:

  • Experience with a programming language such as Stata, R, or Python;
  • Previous research experience, such as through past research assistantships or an independent research project;
  • Strong quantitative background;
  • Experience with Qualtrics survey software;
  • An interest in pursuing a PhD in Economics or a related field.

Start date: as soon as possible

How to apply: please fill out an application here to apply.


Hiring for the Social Science Prediction Platform

The Social Science Prediction Platform is hiring for a pre-doctoral fellow and a full-stack developer. Please see the job adverts below.


Pre-Doctoral Fellowship (filled)

Job description: The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece. 

Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are seeking a pre-doctoral research associate to assist with the Social Science Prediction Platform and be an integral part of the team. The position will be full-time and last for a minimum of one year with the possibility of extension. The Fellow will be based in the Department of Economics at the University of Toronto. Work authorization in Canada, whether by citizenship or an open work permit, is strictly required. Past research assistants working with Eva Vivalt on other projects have subsequently gained admission to top PhD programs in Economics and Political Science.

Duties may include:

  • Designing and testing forecasting surveys for randomized controlled trials in a number of fields, including behavioral economics, development economics, experimental economics, and labor economics;
  • Identifying candidate projects that would particularly benefit from the collection of forecasts;
  • Liaising with researchers at other universities regarding their projects;
  • Cleaning data related to the project;
  • Participating in weekly team meetings with the PIs and other staff.

Minimum required education and experience:

  • Bachelor’s degree in economics, computer science, mathematics, statistics, or a related field;
  • Excellent communication skills in English.

Preferred education and experience:

  • Previous research experience, such as through past research assistantships or an independent research project;
  • Strong quantitative background;
  • Experience with Qualtrics survey software;
  • Experience with a programming language such as Stata, R, or Python;
  • An interest in pursuing a PhD in Economics or a related field.

Start date: as soon as possible

Location: Toronto, Canada

How to apply: please e-mail a CV to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “Pre-doctoral fellowship”.

Deadline: August 9, 2022. Applications submitted after the deadline will be considered on a rolling basis.


Full-Stack Developer

Job description: Profs. Stefano DellaVigna (UC Berkeley) and Eva Vivalt (University of Toronto) are looking to hire a senior software developer with web development experience to take the lead in updating and expanding the Social Science Prediction Platform (SSPP) at scale. The SSPP facilitates the collection and cataloging of forecasts of research results. It uses the Qualtrics API to integrate surveys programmed in Qualtrics on the SSPP interface and a relational database to connect survey responses to project objects and user data such as contact information, demographics, and preferences. The platform generates complex, large data sets which must be kept confidential.

The developer must have work authorization in Canada or the United States.

The SSPP requires significant extensions and updates to efficiently scale up, including:

  • Improving the usability and navigability of the database structure for users with admin privileges;
  • Improving navigability of the project catalog;
  • Improving the visual presentation of survey results;
  • Automated notification to respondents according to their preferences and past completion of forecasting surveys;
  • Ensuring the Qualtrics and Sendgrid APIs continue to function well with evolving  platform features;
  • Enabling users to edit Qualtrics .qsf files dynamically online;
  • Implementing a secure single sign-on system in collaboration with external platforms;
  • Developing a systemized approach to measuring the accuracy of forecasts and providing these accuracy measures to forecasters;
  • Developing a system to automatically pay forecasters according to their performance and ensuring the security of this system;
  • Identifying and merging duplicate accounts; 
  • Building a dynamic “leaderboard” to publicly reward forecasters that provide accurate forecasts.

Required qualifications:

  • Experience with Python/Django/JavaScript (we use Django/Jinja2 as our main framework);
  • Experience with DevOps (we use Docker locally for development and deploy using Heroku, though we would also like the developer to be responsible for migrating the site to AWS);
  • Ability to design and work with REST APIs (e.g., our platform calls Qualtrics APIs to work with survey data, and SendGrid APIs to enable communication with users and stakeholders);
  • Experience with database management, specifically Postgres; 
  • Excellent understanding of architectural frameworks and data warehousing to design solutions for large, complex data sets;
  • Excellent SQL and data manipulation skills;
  • Experience with SSO providers;
  • Experience developing secure payment systems;
  • Back-end development experience, including experience building the back-end of a database or website in which datasets are being queried, manipulated, and transferred to users;
  • Front-end development experience (e.g., building out intuitive user interfaces, SSR, React, cross-origin communication);
  • Excellent communication skills, including the ability to assess and communicate technical challenges and design decisions.

Preferred qualifications:

  • Some understanding of empirical economic/data science methods;
  • Commitment to producing clean, well-documented code (e.g., as evidenced by public GitHub repos).

Start date: as soon as possible

Location:

  • Option to work remotely: meetings during U.S. Pacific Time hours
  • Option to work in Berkeley, CA or Toronto, ON
  • Must have legal ability to work in the United States or Canada 

Competitive salary.

How to apply: please e-mail a CV/resume and expression of interest to Eva Vivalt at eva.vivalt@utoronto.ca with the subject line “developer application”.


Announcing the Launch of the Social Science Prediction Platform!

I have been highly supportive of collecting ex ante forecasts of research results for some time now. Today, I am happy to say that the Social Science Prediction Platform is finally ready for public consumption.

This project, joint with Stefano DellaVigna and with the essential assistance of Nicholas Otis, Henry Xu, and the BITSS team, aims to do several things. Personally, I hope it can:

1. Popularize the ex ante prediction of research results

As argued elsewhere (e.g. here, here and here, as well as in this Science piece with Stefano DellaVigna and Devin Pope), ex ante priors are extremely useful scientifically. Some people already routinely collect them, but to others this is still a relatively new idea.

Personally, I hope that the collection of forecasts of research results becomes somewhat like pre-analysis plans: widespread in certain fields. However, like pre-analysis plans, it may take time for collecting forecasts to really take off. First, we need good examples, templates, and even workshops. This SSPP can be a useful hub to organize these activities. We already have an extensive Forecasting Survey Guide with tips for how to structure forecasting surveys, based on earlier work. We have a template and some annotated examples. We’ve had a workshop on the topic and hope to have more in the future. It is my hope that by gathering these resources together, we can provide a public good that enables others to launch research projects on a variety of topics, such as on meta-science, forecasting, or belief updating, in addition to increasing the value of each individual forecasted study.

2. Solve the coordination problem inherent in gathering forecasts

What coordination problem? Well, if everyone gathered forecasts for all their projects, their private incentive would be to request as many forecasts as possible to maximize their sample size. However, the more requests you receive as a forecaster, the more likely you are to ignore those requests. The platform can help to solve this problem in several ways:

– It can nudge researchers towards sending their surveys to fewer individuals. Pilots suggest that often many forecasts are not needed for a precise estimate

– The platform knows when an individual has already taken a lot of surveys and can direct further surveys to other people instead. For example, someone can specify: I only want to forecast the results of 5 surveys per month. Then we don’t send them more than that. I only wish journals had a similar system whereby they could coordinate across journals and not send anyone too many requests to referee

– We can build in incentives and nudges for people to provide more forecasts. For example, we are planning an incentive scheme for graduate student forecasters, and we suggest that those who want to have forecasts collected for their own projects provide a certain number of forecasts themselves, to be fair.

3. Gather panel data and identify super-forecasters

An advantage that the platform has over individual research teams gathering forecasts for their own independent studies is that we can begin to look across predictions and follow forecasters longitudinally. This could be immensely valuable in identifying super-forecasters and learning more about which types of studies or results are easier to forecast and to what extent forecast ability depends on various forecaster characteristics such as domain expertise.

The first few studies are now up online and ready to be forecast! Over the coming weeks we will post other studies and slowly open it up, so please do get in touch if you have a survey you would like to run. We expect that we’ll continue to learn as we go and would welcome further feedback.