:::: MENU ::::

Disrupt peer review

A great thought from my friend Gautam Rao: “Academic papers should have reviews and ratings like products on Amazon. Peer review shouldn’t be a one time thing.”

I’d really like to see this happen. Citations could cover this in theory but they don’t in practice. Anyone want to do this?

P.S. Gautam is on the job market this year! Check him out!


A change: NY, NY!

I’m happy to share that I will be joining NYU for a post-doc with the Development Research Institute there. I feel very lucky to have the opportunity to interact with such a fantastic group of people and will hopefully have the chance to focus a bit more on research and pushing papers out the door.

This is the institute associated with Bill Easterly and Aid Watch, but it’s entirely independent of my work with AidGrade and you shouldn’t jump to any conclusions. They haven’t been focusing on aid recently, and neither they nor I thinks traditional aid is a large part of the picture; I have always used a broad definition of aid. I’m excited about the opportunity and anticipate some great conversations. Let me know if you’re in the city!


How to give under risk

Part 2 of a series on risk; see part 1 here

A subtle point that I have made elsewhere is that the less sure we are of the outcomes of a development program, the more we should diversify. This is the reason we diversify our investments for ourselves, after all, so why don’t we diversify our investments for others?

Again, I stress that this is a very minor point, because any individual does not need to diversify if, across many individuals, there is already a lot of diversification. Still, it’s worth mentioning because the risk and uncertainty that underlies development programs’ expected outcomes is often glossed over.

A minor criticism of GiveWell (in addition to previous more substantive comments here) relates to this. They recommend only three charities, as they rightly see that there are opportunity costs to giving – we have limited funds to give, so we had better give them to the most effective charities. But, under risk, the most effective option is a diverse bundle. Again, I don’t think this is a major issue, since across individuals there is ample diversification. I just would prefer more honesty about it because I think it really does give the wrong impression about how much we know.

What is more important as an issue, and very neglected in the literature, is the diversity of effects a particular program might have. This comic explains it well. As the world develops more and more, you can easily imagine that the mean effect for a number of programs might go to zero. We need to start paying more attention to the distribution. Here, too, is where meta-analysis can help.


What about risk? Another benefit of meta-analysis

Part 1 of a series on risk; see part 2 here

Risk means many different things to many different people.
When we talk about the risk involved in a certain development program, the first important question is: who is really bearing the risk?

In asset pricing theory, the only risks that matter are risks that cannot be diversified away. If two potential investments have uncorrelated returns, an investor can hedge his or her bets by diversifying his or her portfolio. In terms of development programs, risks can be mitigated by diversification when the success of different programs is not correlated. Since the success of different programs is indeed largely uncorrelated, as they are disperse in space and in their targeted outcomes, an individual donor could diversify the risk he/she would not end up achieving his/her desired outcomes.

However, the way we normally think about development, the donors are not part of the goal. It can be okay for an individual donor to have funded an unsuccessful program so long as, combined, the aid programs that are funded are successful (though again this depends on a number of other things such as the extent to which unsuccessful programs hurt intended beneficiaries or others).

In other words, donors can hedge their own risks by diversifying the portfolio of programs they donate to, but this is irrelevant to the beneficiaries, who do not receive a diversified portfolio of programs. When we think about risk, we should instead pay attention to whether individual programs themselves are risky.

Yet whether an individual program run by one particular NGO in one particular place at one particular time is risky is precisely the thing on which we have the least information. Even if we were so lucky as to have an impact evaluation of the program so that we knew its past success, and even if we were so lucky as to have that impact evaluation capture the estimated effects on different parts of the population rather than just the mean effect, this still wouldn’t tell us how risky that program was. You can’t quantify the dispersion of the estimates of the mean, for example, when you only have one mean.

We have to resort to analysis of a more general type of program. When you repeat the program, you are not repeating exactly the same program – if nothing else, the intended beneficiaries have changed, having benefitted or been harmed by the previous program. Still, it seems reasonable to use past data as evidence on which to base future policy (in a Bayesian approach, it should still cause us to update our priors, even if it is not fully predictive). And at least when you observe multiple instances of the same program being rolled out, you can measure the variance across programs, or the standard deviation, a more typical measure.

This is another benefit of meta-analysis. As a side perk, you get to see how much results vary, under which contexts, and calculate the coefficient of variation within interventions. Some types of programs do systematically vary more in their results than others. I have a working paper on this and am happy to discuss further.


P.S. to Reinhardt-Rogoff post

A couple of people asked: why am I calling for a pre-analysis registry for non-RCTs? Can’t people look at the data for non-experimental studies and then “game” the pre-analysis plans, essentially writing them after the analysis?

Not necessarily. There can still be substantial lag time before you get the data for non-experimental studies, as well. I’d say I’m averaging more than a year to get non-experimental data… even including the times when the data exist out there and I just have to repeatedly ask for them! Mostly, though, I’m thinking about the large number of development economics studies which use quasi-experimental methods but otherwise face the exact same lags as RCTs do. There’s got to be something for them.