Making pre-registration work

Here is link to the COMPARE project, an incredible, though depressing, project designed to check if clinical trials are following their preregistered analysis plans when published.

Pre-registration is a great technique to help draw a bright line between planned and exploratory analyses.  It’s not that there is anything wrong with exploration, but it is essential when presenting results that findings made through exploration are clearly presented this way.  It is an amazing thing to test a drug and find it has exactly the predicted effect on a specific dependent variable.  It is interesting but much less amazing to find that out of forty variables assessed the drug had an interesting effect on two of them.  The really crucial thing, though, is not to have one type of analysis masquerading as another: not to dress up a finding fished out of the sea as exactly what one was expecting.

Pre-registration makes your analysis plan publicly verifiable.  But that means the public must actually pay attention.  The Compare project does just that.  The researchers examined 6 weeks worth of clinical trials published in top medical journals.  For each trial they compared the published article with the preregistered analysis plan.  They flagged places where planned analyses had been silently dropped and where new analyses had been introduced without flagging them as exploratory.  The results were sobering: out of 67 trials checked, only 9 were perfectly reported; all the others added or omitted analyses without flagging these changes.  Overall, over 300 new exploratory analyses were reported without being flagged as exploratory and over 3oo planned analyses were silently dropped from the manuscripts.  That’s terrible!  Even worse, when the researchers pointed out these issues to the journals, many of them had negative reactions that indicated a lack of commitment or understanding about the importance of drawing a bright line between exploratory and planned analyses.  Ugh!

Click the link to see the results in detail.  In particular, check out the blog section which tracks the response to the research including email correspondence with the journal editors.  It is really worth reading.  The take away, it simple: we need to work harder at ensuring preregistration is used and at COMPARIng manuscripts with preregistration documents to ensure researchers are reporting results as they should.