Why Most Published Research Findings are False
Why is it that it’s considered noble to ‘reason’ with someone, but’rationalizing’ isconsidered to be dishonest? How is an ‘excuse’ different from a ‘reason’? The words all describe the same activity, saving the appearances, but the various forms connote differing degrees of relative ‘truthfulness’, something which is hopefully determined independently of ‘reason’.
The modern day cult of reason holds that thruthfulness is a function of lots of good, repeatable observation and data. It’s not a bad assumption, and might even be proven true on the day that we have access to all of the universe’s information and have constructed a logical system with no flaws. But in the real world, it’s pretty darned easy to get bad data, and easy to arrive at the wrong conclusions when the data is good (or not verifiably bad).
Throughout my career, I’veworked with a variety ofanalytical tools used to justify decisions. A minor obsession of mine has been to observe how often business decisions are made on flawed data and results. The first time I observed this was very early in my career, when I discovered some flaws in a spreadsheet model being used to justify several million dollars expenditure. The flaws were small, but cascaded to the final result to completely change the outcome’s profitability and trend lines. I’ve since seen studies which show that 90% of spreadsheet models used in business contain errors, or that more than half contain errors which significantly impact the result. Feed bad data into these models, and the picture just gets worse. The same kinds of errors can be found in models created with other analytical tools.
Now Marginal Revolution reasons that more than half of published research findings are false. It makes sense, really (and I believe he gives economics too much of a pass, since economics suffers from different problems than medicine, such as dubious repeatability, inability to control important variables, and wide contextual variation).
I’m not arguing that reason and analysis are useless, however. They are incredibly useful tools. But the level of blind faith placed in these tools, and the lack of healthy scientific skepticism, often have me thinking of the emperor’s new clothes.