If I didn’t know better, I’d say Gary Gutting was reading my blog: http://tinyurl.com/cn99foj Professor Gutting discusses the exact problem I described in a recent blog – the proliferation of scientific studies in the media – and the fact that these studies are misrepresented. Gutting’s suggestion for a coding system is well received. As he points out, there are different types of studies – ones that suggest a correlation, and those that establish a causal link. In many cases, the media fails to distinguish between the two.
Krugman’s take on the R&R debacle: http://tinyurl.com/cz8yv45
From Tetris to coffee stains, an interesting article on the fascinating topic of universality: http://tinyurl.com/b9t28gs
Here’s an article from the Chronicle on the problems facing the fashionable theories of today’s physics. Again, refer to Peter Woit for a full treatment of symmetry and string theory. http://tinyurl.com/aplg9j7
Oh boy. In the last post, I discussed the importance of replicating research as a mechanism in the scientific process. Replication is especially important when the original research informs policy decisions and influences political actors. Well, it looks like a seminal paper (RandR) in economics by Reinhard-Rogoff (followed by their book (“This Time is Different: Eight Centuries of Financial Folly“) contains some outcome-changing errors.
You might be thinking…well, so what? Some nerdy economics paper turns out to be off the mark. Well…the problem here is – when it comes to debating economic policies for countries (think U.S. and E.U.) – the paper in question is the most influential in the aftermath of the global economic crisis. Since publication, it has been cited about 450 times in related work. (see here for a fuller explanation of the issues with the R&R work). The three main issues that distorted R&R’s results were: selective exclusions, unconventional weighting, and a coding error. The coding error is really quite unbelievable. Basically, it’s a spreadsheet error. Therefore, the key research that provided the basis for global moves toward austerity turns out to be partly based on someone messing up in Excel.
What’s interesting is how theories in social sciences go from paper to book to influencing policy with very little attempt to replicate the original results. Obviously, not the first time this has happened. Still, scholars need to wake up and smell the coffee.
This op-ed by Lawrence Krauss explains the amazing picture of the universe assembled by the Planck satellite. It’s a good read. As some may know, Krauss is no stranger to controversy. In the past year, he has been at the epicenter of the feud between scientists and philosophers about the existence of nothing and the value (or lack thereof) of philosophy. (See post Science Vs. Philosophy). In Krauss’ current piece, he makes a couple of ironic comments and takes the obligatory jab at philosophy. Speaking of some of the anomalies in the data, he writes:
If history is any guide, most of these anomalies will disappear. It is an unfortunate facet of science reporting that it isn’t often made clear that most anomalies in experiments tend to go away, just as most theoretical ideas turn out to be wrong.
Indeed. It would be refreshing if physicists would finally come to terms with the fact that String Theory is one such theory. In fact, as far as String Theory is concerned, it is even a worse state of affairs. As Peter Woit of Columbia University points out, respectable theories can turn out to be wrong (because they are testable). Alas, String Sheory is not even wrong (see Woit’s Book). It is not a testable theory now…and there is good reason to believe it will never be tested. So, for all the print spilled on this theory…not to mention the research dollars, someone of repute should step forward and say this theory is dead. (By someone of repute I mean one of the flag wavers who advanced this theory and popularized it for the public). The irony here is that Krauss and his cohorts are leading the attacks on philosophy and religion. And yet, String Theory was worked on for years more as a matter of faith than good science. So, it would be helpful to remember that scientists are susceptible to collective cognitive episodes of irrationality. And Krauss goes on…
Instead of attributing significance to potentially strange results, it is the business of science to try and prove them wrong before we blindly move forward. Skepticism is the business of the day, and it is wise to remember this next time you read an astounding discovery claimed in the press.
Yes. And not only should we cast a skeptical eye on discoveries, we should also be aware that the medical research community has failed to police itself when it comes to publicizing the results of their studies. Each week we read about some study that claims some benefit from consuming this or that…or abstaining from one thing or another. Other studies focus on the benefits of a certain drug or intervention for a variety ailments. Despite the attention afforded these studies in the media, most are either proven wrong or demonstrated to have exaggerated the effects of the independent variable. Want some numbers? A 2005 study on this topic was conducted by John P. A. Ioannidis, MD (Harvard) examining how often highly cited clinical research studies were either contradicted, proven less than reported, confirmed with other studies, or unchallenged. Here are the results from the study:
Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with “negative” results.
It’s 2013 and I guarantee nothing in the medical research community has been done to change this sad state of affairs. Let’s put these numbers in perspective. Highly cited studies get reported in the media and their conclusions adopted by many doctors and practitioners. Of the 49 studies Ioannidis examined, less than half were replicated! Replication is the controlling mechanism in all of science! And aside from the fact that 16% were contradicted, 24% of these studies had never been challenged. I suspect the rate of contradiction would go up if the remaining 24% went under some scrutiny. Here is the Ionnidis study: JOC50060
So, I agree with Krauss, skepticism is the business of the day. And it applies to all fields. Read about how Vitamin E increases your ability to solve differential equations? Don’t run out and buy a barrel of sunflower seeds. Read about how pomegranates gives you everlasting health? Don’t run out to Whole Foods just yet. The bottom line here is that most of what is reported by medical research needs to be taken with a grain of salt. But just a grain..because too much salt will kill you.
High-res image of the Planck picture of the universe.