Briefly: BITSS is highly necessary.
As mentioned in the new transparency series on CEGA’s blog, there are a lot of problems in the discipline today. Focusing on interaction terms or particular subgroups is one way of increasing the odds of obtaining the elusive 5% significance level, so a lot of people do it, but this overstates the results’ true significance. Something can always appear significant by accident, and tests of significance are only legitimate if they are defined beforehand.
Because of this issue, AidGrade has been forced to take a very conservative stance with regards to the values it collects from studies. In the absence of pre-analysis plans, we have avoided collecting interaction terms and have also focused on results containing few controls (another way that people can lie with statistics). This is obviously a shame, because there is a real sense in which these terms could be important. However, in the absence of pre-analysis plans, this is the best we can do in order to avoid fraud.
Here’s hoping the discipline will change with all the new attention brought to bear on this topic!