:::: MENU ::::

Using machine learning for meta-analysis

AidGrade is starting to use machine learning to help extract data from academic papers for meta-analysis. This is a big deal – meta-analyses tend to go out of date quickly because data extraction is such a time-intensive process and new studies are constantly coming out at an ever-increasing rate.

AidGrade will use its existing database of impact evaluation results to help build and validate models. For each extracted piece of information, it will also generate a probability that the information is correct.

At the very minimum, this will reduce the amount of time it takes to identify key characteristics of studies, such as where they were done and which methods they used. It is also the only way to ensure that meta-analyses are perpetually updated as new studies come out. Given that the methods should be scalable to much of economics, education, and health (think of a ScienceScape (update: now known as Meta) for meta-analysis – they have catalogued 25 million studies, a number which one would definitely need machine learning to process!), it will build this tool in a general way so that its results can be used to inform policy even in developed countries.

To support this, AidGrade has a new crowdfunding campaign. Please share and contribute.

How much does an impact evaluation improve policy decisions?

Thanks to excellent feedback, I’ve extended my generalizability paper to include discussion of how much an impact evaluation improves policy decisions.

Results, in a nutshell: the “typical” impact evaluation (of a program with a small effect size, compared to an outside option that also has a small effect size) might improve policy decisions by only about 0.1-0.3% (of a small amount). If the outside option is much different (say an effect size of 0) and it is one of the earliest impact evaluations on a topic, this can go up to 4.6%.

There are a lot of caveats here, chief among them that an impact evaluation provides a public good and many people can use its results.

Nonetheless, personally, I find this sobering. I don’t think we’re usually in that best case scenario. These aren’t the results I want, but they are the results I get.

Veganism, Part 2: Psychological biases and food choice

Part 1 here.

I’m vegan for all the usual reasons. If you think you have an objection, I encourage you to look it up and see if someone’s addressed it here — they have a lot of common responses.

That’s not the topic of this post. This post is about how to make it easier, building from the psychology literature. I suspect the main thing holding people back from going all the way (or going vegetarian) is not that they don’t agree it’s a good thing, but that they feel it is hard. It would be silly to think that our food choices are immune to psychological biases, so it’s worth exploring why it might be easier than you think.

First, I think projection bias plays a large role here, in that people don’t realize how much their tastes will change, and tastes are mostly endogenous. There’s a reason that if you ask people what their favourite foods are, they will often point to food they ate when they were a child or food from their hometown. For most people, the example of switching between 2% and skim milk also seems to resonate: whichever one you were previously drinking, if you switch it will taste disgusting at first, but in time you get used to it, and in fact if you were to try to switch back you would find your original choice disgusting.

Recognizing that every choice you make is not just a choice about what to eat today but a choice about what you will want to eat in the future helps. It provides a very motivating and positive framing.

Another bias people have is that they think of all the things they know that they like to eat, and they have a hard time picturing what they would eat if they went vegan/vegetarian. A nice analogy I heard: people go around with a box their whole lives, and every time they find something they like, they put it in the box. When they get to the point of considering veg*nism, they compare their box of stuff with a new, empty box of what they would eat if they were veg*n. But it’s not a fair comparison, because they have had a long time to accumulate all those things in their box, and if they went veg*n, in time they would find other things to add to the veg*n box (although there would be a temporary cost).

Another thing that I think can be helpful psychologically is reframing the action so that instead of reducing meat, one adds vegetables/etc. If you add enough good, tasty stuff, you’ll be more likely to crowd out the rest.

The bigger problem, my fellow vegan economist friend Josh suggests, is information aversion. Also, the tendency for people to be inconsistent and biased when it would help them avoid cognitive dissonance. People are less likely to assign farm animals moral weight when they are hungry or after they were served meat. Josh, Emiliano and I are planning more research in animal advocacy.

Why I am vegan

It is near U.S. Thanksgiving, so a good time to discuss why I am vegan. This will be part 1 of 2 posts on veganism.


The food industry is terrible to animals. There is no way to deny that farm animals are sentient — we know this as well as we know that other humans are sentient. Most farm animals grow up in hellish conditions; “cage-free” is not necessarily much better; “humane” certifications are more like recommendations and are not actually binding if you read the “requirements” (e.g. chickens should enough room to “stretch their wings” — this is the same thing that the previously-discussed cage-free hens have, and anyway individual farmers can be allowed a higher density); even if humane certifications had any teeth, I do not agree with the exploitation of another living, sentient being.


Meat requires more inputs, such as water, since you need a large number of plants to raise an animal.


Some animal advocates do not like it when health is mentioned as a reason to go vegan, because health fads come and go. Still, I think it needs to be mentioned because so many people seem to think that it is a “me or them” situation. Veganism can be as healthy as any other diet. Meat-eating has a particularly strong causal connection to heart disease, so if it runs in your family it makes even more sense to avoid it. You can also be vegan and low-carb. Personally, I try to avoid FODMAPs for GI issues — no problem.

Other arguments

I often hear the same set of complaints (“our ancestors did it”, “plants have feelings”, …). If you think you have an argument against veganism, please check this list of counterarguments first. Some of their comics are fantastic!

Other words

More than “vegan”, I am “liberationist” — that is to say, sometimes people use “vegan” to refer only to a dietary choice rather than a philosophy, and I would like to be clear that I do not agree with the exploitation of other sentient beings.

Don’t use the passive voice in scientific writing

This is a hideous norm that must be stopped.

I remember specifically being taught in school to use the passive voice when writing “science”. The problem is that it disguises what actually happened, and implementation details matter. That studies can be done in a vacuum might hold true for a few narrow disciplines but is generally misleading. It also constrains the future use of the data.

As an example of how paucity of details can hurt, in AidGrade’s dataset, we have the year the study was published, or the year of the working paper if the paper was not published. Collaborators at Stanford are interested in matching Demographic and Health Survey (DHS) data to the geocoded interventions to investigate the long-run effects. For this, we need the year the intervention itself began and ended. More than 25% of the studies for which this was coded turn out not to have this information.

Worse, when how a program was implemented matters, many studies do not report this information in sufficient detail.

I would like to have a “how to write well” book to recommend to people. Any suggestions?