:::: MENU ::::

The limits of meta-analysis

With all that I have been promoting the Kickstarter, you would think I wouldn’t highlight the limits of meta-analysis.

But progress depends on being honest about the strengths and weaknesses of any method. So let me tell you a story about when a meta-analysis didn’t really help much. This story is essentially the opposite of a meta-analysis, being simply an anecdote.

Young as I am (late 20s), I hurt my back. Badly. I hurt it 4 times in the past year. The most recent time, I was simply trying to walk down the street when it struck, bad enough for me to almost pass out on the street (losing most of your senses but retaining consciousness is a strange feeling).

Things got a lot better. But later on, about to go to Kenya for an impact evaluation planning workshop, I was a little nervous at the prospect of having to sit for a prolonged period of time, since that’s what caused my back to go out in the first place. So I returned to my physical therapist and she suggested I try “dry needling”.

I hadn’t heard of the term before, and to be honest I didn’t look into it too much. This particular physical therapist’s office had already proven its worth in gold, and I had great insurance, so why not give it a shot.

It wasn’t done to treat pain but to stretch out my hamstrings a bit, because your hamstrings can pull your back out of place. Before having the dry needling done, my right hamstrings were always tighter than my left. Every single time I stretched, for months, there was a visible difference — I had over 100 observations.

The physical therapist only dry-needled my right hamstrings, not my left. And afterwards, my right hamstrings, for the first time, stretched farther than my left. I wasn’t anticipating that, and I was honestly surprised and had to take a moment to attribute the cause, so I don’t think it was a placebo effect.

It seems pretty darn causal to me!

Yet I later heard that no firm conclusions have been drawn about dry needling from meta-analysis.*

Without looking at the paper, I could think of two main hypotheses to explain this:

1) Perhaps it has mixed effects and I just got lucky;

2) Perhaps the studies that went into the meta-analysis were of poor quality.

It turns out the reason for the lack of firm findings is much more mundane: they actually do not find many good-quality studies to include (see, for instance, the number of studies cited in Figure 3 on page 25). Honestly, when they get down to using one paper at a time, it stretches the definition of “meta-analysis”.
Still, the episode highlights a major problem meta-analyses face — they are only as good as the studies that go into them.

They also seem to generally put undo emphasis on the mean, when we may be just as interested in other moments. Impact evaluations are often guilty of this as well, reminding me of this comic.

Despite this, you can be sure that if we ever do get a better sense of the general effectiveness of dry needling and the conditions under which it works, it will be because a meta-analysis based on more good studies will have synthesized the data.

* That paper looks at low back pain rather than hamstring flexibility, but there is even less evidence for hamstring flexibility.

Important note: Definition of “aid”

I use a very broad definition of aid.

Because we all know that when it comes down to it, “traditional” aid contributes little to development.

“Aid” for me is anything that improves well-being (well-being is also not the same thing as income). Could a political protest be aid? Yes. Could improving investment climate be aid? Yes.

Why not just call it “development”? Things can develop on their own, and I would like to imply agents (of the “do with”, not “do to” kind). I also think it’s a good idea to nudge those with a more traditional view of aid to consider a wider range of possibilities. It’s a reclaiming of the term.

Honouring others

A lot of smart people work in development.

Those who blog include (but I am sure this list is incomplete!):


What am I missing? Because I am certain to have missed some good ones.

There are also many other blogs I love that unfortunately don’t quite fall into the category of development. In fact, I tend to love these ones best, to learn outside my area.

The Kickstarter

The behind-the-scenes story of the Kickstarter is that these meta-analyses were going to happen anyway. I get involved in things, and somehow I ended up getting involved to the point of having my apartment be used as an office for students while I’m off at work. Typical.

But the more you do, the more it gets difficult to coordinate things without money being involved.

I believe the cause is good. I believe it is only one small part, but a useful part. Packaging it into a book is only for the benefit of the Kickstarter; it wasn’t the goal.

The development community is an intensely cynical and skeptical community — through experience. Many people do development/aid/charity to make themselves look good. If anything, I think this Kickstarter threatens to make me look bad. It looks blatantly self-promotional (though it is all for the student team), but I’m just following the format of other Kickstarters and I don’t know how else to sell it without putting myself out there. I’m better situated to do that than the student team, after all.

Fortunately, throughout life, I’ve noticed that if I’m open and honest and cheery, people somehow end up liking me. So here goes nothing.

Hello, world: Why I’m blogging

I’ve decided that in this day and age, I need a “platform”. For a shy person, this is horrible. But I care about the issues, so c’est la vie.

Why I’m blogging:

1) Because I hate putting myself out there, actually. Does that sound strange? I hate it, so I’ll do more of it? I figure a blog is a good way to defend myself and explain my views in more depth than I can via twitter, e-mail, etc. I don’t like giving short shrift to more complicated issues.

2) I have something to say. I have spent a large part of the past 10 years thinking of how best to help the world, so I’d like to think that occasionally I have something important or useful to say.