by Stephen T. Ziliak and Deirdre N. McCloskey
How the most important statistical method used in many of the sciences doesn’t pass the test for basic common sense
“A group of mathematicians has been trying for years to have a core statistical concept debunked. Now, the Supreme Court might have done it for them.”
– Carl Bialik, The Wall Street Journal, April 2, 2011, p. A5
Praise for the Book* Amazon.com Author Page
*Best-seller at The University of Michigan Press, 2008-present
Reviewed (as of Fall 2014) in approximately forty (40) of the leading journals, including Science, Nature, Journal of Economic Literature, and Notices of the American Mathematical Society
“With humor, insight, piercing logic and a nod to history, Ziliak and McCloskey show how economists–and other scientists–suffer from a mass delusion about statistical analysis. The quest for statistical significance that pervades science today is a deeply flawed substitute for thoughtful analysis. This hollow pursuit, kept alive by mechanical, conformist thinking, has led to grave and obvious errors. Yet few participants in the scientific bureaucracy have been willing to admit what Ziliak and McCloskey make clear: the emperor has no clothes.”
—Kenneth Rothman, Professor of Epidemiology, Boston University, Distinguished Fellow, Research Triangle Institute, and founding editor of Epidemiology
“The book is a model of scholarship, transparent in its method, wide-reaching in its disciplinary expertise, and highly literate, including occasional haiku poems and humor such as, ‘If the variable doesn’t fit/you may not have to acquit.’ The authors convincingly argue that environmental quality, jobs, and even lives are at stake.”
—M. H. Maier, Choice
“McCloskey and Ziliak have been pushing this very elementary, very correct, very important argument through several articles over several years and for reasons I cannot fathom it is still resisted. If it takes a book to get it across, I hope this book will do it. It ought to.”
—Thomas Schelling, Distinguished University Professor, School of Public Policy, University of Maryland, and 2005 Nobel Laureate in Economics
“One ingredient, which during the course of the twentieth century has permeated all of science to the extent that it is nowadays recognized as indispensable, is mathematical statistics, which helps researchers distinguish between pattern and noise and to quantify how much confidence in our conclusions the data warrant. There can hardly be any doubt that this development has been of immense benefit to science. All the more interesting, then, that two prominent economists, Stephen Ziliak and Deirdre McCloskey, claim in their recent book The Cult of Statistical Significance [ZM] that the reliance on statistical methods has gone too far and turned into a ritual and an obstacle to scientific progress. A typical situation is the following. A scientist formulates a null hypothesis.” —Olle Häggström, Notices of the American Mathematical Society
“If not Fisherian significance, what should be the Holy Grail of statistics? Ziliak and McCloskey . . . answer: “Oomph.” We should identify quantities that matter and measure them, not merely determine whether they can be distinguished from the null (meaning no effect) at some predetermined likelihood level. The validity of this point I take to be virtually self-evident. Yet statistical tests that ignore quantity remain pervasive, as the authors demonstrate through quantitative analyses of the contents of some very prestigious journals of economics, psychology, and medicine.”
—Theodore Porter, Science
“To be sure, there were dissenters, mostly Bayesians, to Fisher’s rarefied yet influential view of science. Most heroic of them all was William Sealy Gosset, a student of Karl Pearson who pioneered the treatment of hypothesis testing as a species of decision-making. As director of the brewing labs at Guinness in the early 20th century, Gosset introduced the “loss function”, which given the available evidence calculates the cost of accepting a hypothesis visà-vis potentially better hypotheses that might require further testing. For Ziliak and McCloskey, Gosset provided a rigorous take on what “significance” means in policy-relevant research.” —Steve Fuller, Times Higher Education
William Sealy Gosset (1876-1937) aka “Student”
“A clear trade-off: how much confidence [in a result] is “enough” depends on the costs of further research and the benefits of extra precision. Ziliak and his co-author Deirdre McCloskey argue in The Cult of Statistical Significance that most academic disciplines have forgotten this trade-off . . . A sharp line for statistical significance makes no sense, and it has a cost.”
—Tim Harford, The Financial Times
“[Steve Ziliak and Deirdre McCloskey] explain to us why the misunderstanding of statistical significance has lead to bad government policy making and how one particularly famous brewery employed the technique to improve the pints we enjoy today.”
—Tim Harford, BBC
“[S]uppose a regression is run to estimate the demand function for gasoline and the price elasticity of demand is estimated to be –0.80 with an estimated standard error of 0.50. What conclusion is to be drawn? It might be concluded that since the corresponding t-statistic is just –1.6 (= –0.8/0.5), price is insignificant; end of story. McCloskey and Ziliak would claim that such a conclusion is a mindless use of a statistical rule-of-thumb and doesn’t begin to address issues related to economic importance.” —Saul Hymans, National Bureau of Economic Research, Journal of Economic Literature
“Persuading professionals that their procedures are wrong is a long and lonely task. McCloskey, joined later by Ziliak, has been conducting such a crusade against the misuse of significance testing for over 25 years. This book presents their argument, gives lots of examples of the adverse consequences of misuse, and provides some history of the controversy, which dates from the origins of mathematical statistics.”
— Ron P. Smith, Journal of Economic Issues
“The Cult of Statistical Significance has virtues that extend beyond its core message. It is clearly written and should be accessible to those who have neither formal training in statistics nor a desire to secure any. It is full of examples that illustrate why it is the strength of relationships and not their statistical significance that mainly matters.”
—Richard Lempert, Law and Social Inquiry
“Despite appearing to be a book of limited appeal – it is after all a book that looks at a set of statistical techniques – it is one that has immense social implications. We live in an age where ideologies have largely been cast aside and instead we are governed increasingly by a class of politicians and civil servants who aim for ‘evidence-based’ policy-making. When that evidence is based on statistically significant results that ignore any quantification of results then we all have reason to pay attention.”
—London Book Review
“Nevertheless, the message remains: Even employees of major U.S. statistical agencies might take statistical significance for granted. After all, we and other statistical practitioners and data disseminators know all about estimate formulation and sample error. We can analyze data and present our survey results and research findings to the public, providing valuable information about our economy. Relatively few of us, however, know the history of significance analysis, the controversy that surrounds its use, and the “substantive” strength added by considerations of power and other analytical methods.
Cult’s strength is that it fills that void….and then some.” —Bruce Bergman, Bureau of Labor Statistics, New York Office, Monthly Labor Review
See also: Brief of Amici Curiae by Statistics Experts Professors Deirdre N. McCloskey and Stephen T. Ziliak in Support of Respondents (vol. No. 09-1156, pp. 22). Washington DC: Supreme Court of the United States. With Edward Labaton et al. Counsel of Record (Ed.), on adverse effect reporting and statistical significance, Matrixx et. al. v. Siracusano and NECA-IBEW Pension Fund, filed Nov. 12, 2010: pdf McCloskey and Ziliak Supreme Court brief.
See also: Late-Breaking Session, Supreme Court Finds Statistical Significance Is Not Necessary for Causation_Matrixx v Siracusano Student v Fisher_Ziliak PDF, Joint Statistical Meetings, 2011, Miami Beach; Supreme Court oral arguments on statistical significance Jan 2011, the Supreme Court decision against statistical significance March 2011 and interviews that Ziliak did with Tim Harford (BBC) and Carl Bialik (Wall Street Journal).
