Reuters reported on March 28, 2012 that many cancer ‘discoveries’ were inaccurate and irreproducible according to a former researcher at Amgen, Inc. This researcher, C. Glenn Begley, chose 53 findings that were published by reputable researchers in top-tiered journals. He tasked his research team to reproduce those findings in the lab. Out of the 53, the team managed to replicate only six of the studies.
One possible reason for this?
According to the Reuters story, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies.
“We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”
What the heck?! A researcher reported on something he saw one time?!
That is so wrong!
Scientists aren’t trained this way. Or at least responsible scientists aren’t.
We’re taught to repeat (repeat! repeat!) experiments several times to be sure that the results are “real” and not a fluke. That it’s a real and observable phenomenon and not something we’ve inadvertently built into the system.
I can’t speak for other scientific disciplines like chemistry or physics, but in biology, the systems we’re studying don’t always cooperate nicely. We may have to repeat an experiment many times because there might be a lot of variability in the measurements we’re taking.
Even when we use supposedly pure cell cultures, we’ll see some “bounce” in the results. And when you’re talking animal studies or clinical studies, it’s just that much more complicated to see a pattern in the data. Often the pattern is so difficult to see that statistics are brought in.
Scientists are also trained to approach the problem from more than one angle, to show the phenomenon in more than one way. Again, this is to be sure that the results are real and not something to do with the way we’re testing our hypothesis. So, for example, say we want to show that a drug is not harmful to the cells. We might add the drug to the cells and merely observe them (documenting it in photographs, of course). But that’s not enough to say the drug didn’t kill the cells. To say this, we would have to perform other techniques, maybe look at certain proteins that become expressed on the surface of a dying cell or look for damage to the DNA or look at the health of the mitochondria (the powerhouses of the cell – are they still cranking out the power or did they shutter up the factories and move on?). By coming at the question from various angles, the answer is closer to the truth.
Does an experiment work perfectly every time? Not always, even though we try to do things exactly the same way every time. There’s human error, instrument error, misalignment of the planets (for those who are a little on the superstitious side) and, of course, mother nature herself. Any or all of these things can make an experiment come out slightly different each time it’s done. But the pattern should be there.
Do scientists show their “best” data? You bet we do! We find the best photo that shows what we saw (every time, not just once). We show the best graph of the results we saw (every time).
Do scientists show results they’ve only seen once? If they’re ethical and honest researchers, absolutely not! Unless, of course, they explain that in the published article.
I am appalled that this researcher (the one who confessed to Begley) would act in such an unprofessional way.
Is there this much pressure to publish that the scientist would pluck one unreproducible result to show in his publication just because it fit his hypothesis?
All I know is that scientific conduct like that casts doubt on the scientific community and erodes public confidence in the scientific process.
So, come on all you scientists out there: avoid the shortcuts and do the good science I know you can!
biologyze 