Go read this bit of news published in Nature recently (especially the comments). The quick recap is that a social psychologist, Ap Dijksterhuis has an effect, published over a decade ago, that others have yet to replicate. Researcher’s are skeptical, some replication attempts have failed, he stands by the effect. There's been a lot of back and forth.
The article leaves me feeling a little uneasy. Not just the looming issue of unreplicability, but how we scientists are going to deal with it.
“[Dijksterhuis] adds that social psychology needs to get more rigorous, but that the rigour should be applied to future, not historical, experiments.”
He seems to be doing his best Mark McGwire impression: I’m not here to talk about the past (McGwire was a baseball player accused of cheating whose response was basically to suggest we all forget it and move on).
The article ends with this stinger, a quote from an email sent by one of the skeptics of the effect.
“refusal to engage in a legitimate scientific conversation … invites the interpretation that the believers are afraid of the outcome”
I don’t want to give the impression that I don't think fraud and false positives are issues. This article in Nature, which is whether you like it or not, a flagship journal of science research. Maybe I'm reading between the lines too much here but the article seems to end with:
They’re not sayin’, they’re just sayin’.
Not to long ago baseball was in a similar situation regarding cheating with steroids. There is clearly some bad behavior going on. We're not totally sure what to do. Ideas range from testing everyone all the time to doing nothing. Baseball went through a phase where everyone was suspect if there performance was a bit too ... unexpected. Perhaps that is where we are now?