Why Most Published Research Findings are False
Why is it that it’s considered noble to ‘reason’ with someone, but’rationalizing’ isconsidered to be dishonest? How is an ‘excuse’ different from a ‘reason’? The words all describe the same activity, saving the appearances, but the various forms connote differing degrees of relative ‘truthfulness’, something which is hopefully determined independently of ‘reason’.
The modern day cult of reason holds that thruthfulness is a function of lots of good, repeatable observation and data. It’s not a bad assumption, and might even be proven true on the day that we have access to all of the universe’s information and have constructed a logical system with no flaws. But in the real world, it’s pretty darned easy to get bad data, and easy to arrive at the wrong conclusions when the data is good (or not verifiably bad).
Throughout my career, I’veworked with a variety ofanalytical tools used to justify decisions. A minor obsession of mine has been to observe how often business decisions are made on flawed data and results. The first time I observed this was very early in my career, when I discovered some flaws in a spreadsheet model being used to justify several million dollars expenditure. The flaws were small, but cascaded to the final result to completely change the outcome’s profitability and trend lines. I’ve since seen studies which show that 90% of spreadsheet models used in business contain errors, or that more than half contain errors which significantly impact the result. Feed bad data into these models, and the picture just gets worse. The same kinds of errors can be found in models created with other analytical tools.
Now Marginal Revolution reasons that more than half of published research findings are false. It makes sense, really (and I believe he gives economics too much of a pass, since economics suffers from different problems than medicine, such as dubious repeatability, inability to control important variables, and wide contextual variation).
I’m not arguing that reason and analysis are useless, however. They are incredibly useful tools. But the level of blind faith placed in these tools, and the lack of healthy scientific skepticism, often have me thinking of the emperor’s new clothes.
September 6th, 2006 at 12:54 am
The “cult of reason” is like the “cult of money” and of course like religion. The allure is that there is a supremely simple value system against which any action can be judged. Reason == having one, money == more is better, religion == be a good little boy/girl. How convenient and free from any pesky freedom/responsibility! Any potential action can simply be mechanically deconstructed to obtain an answer as to whether or not it should be carried out. In a closed social system, values (ends) are the unconscious-because-universal substrate — only the means remain to be resolved.
Now if something you do comes under question, it’s not your fault, it’s the system’s! You might as well blame the sun for shining! That is when reason becomes rationalization, rapacious greed becomes stockholder interest, and religion becomes an excuse to sin. (I am only human!) In the end what is important is that you are perceived to be on the right side of the fence. That you are “in” with your cult, and if your cult has negotiated a treaty (such as corporate person-hood), by extension society at large. Because then your license to ill remains intact.
“I am right.” But wherefore, dear cog?
September 7th, 2006 at 7:29 am
Snuggles:
Right on. Systems are a way for us to blame the victim (people are poor because they’re stupid), avoid responsibility for our brother, and so on. There is something to be said for economies of scale; but any system just amplifies the intentions and deceits of it’s participants — and thus is only as “good” as the people therein.