Analyzing Statistics

This is a guide for writing and discussion about the use of statistics in news articles and other mainstream media. Use the questions below a starting points for your analysis of how numbers are used to report (and sometimes obfuscate) the news.

What kind of data are cited?

  • What sort of values are reported? Are they absolute numbers (e.g., 1,555 widgets), percentages, averages, rates? Make sure you know what kind of numbers you’re looking at.
  • If an average is reported, what kind is it?
  • How are the data displayed? Within the article itself, in a table, in a graph (or even map), or a combination of these?

Where did the data come from?

  • Are sources mentioned for the cited statistics?
  • If not, is this problematic for interpreting them, or is it safe to assume they’re credible? If so, how credible are the sources? (Are they peer-reviewed, for example? Do they have a possible agenda that could adversely affect the data’s validity, and how much do we need to account for that?)
  • How much information is provided in the article to allow you to track down the original source of information? (The more, the better.)

How were the data collected?

  • What kind of information are you provided about the methods behind the data collection? Is anything noted about methods that were employed to ensure accuracy and reliability?
  • If you’re looking at poll numbers, how was the poll conducted?
  • If a study based on samples, do you know if the sample is representative of the target population? In other words, is it randomly selected and of sufficient size?

How good are the data?

  • Are possible sources of error noted? If not, can you speculate on what error sources might exist?
  • What is the margin of error for the reported numbers, and how important is that to interpreting their meaning? Can you tell if the data are statistically significant?
  • Any possible sources of bias in the data, either acknowledged or unacknowledged?

How meaningful are the data?

  • Are the statistics central to the article’s purpose in conveying information, or marginal? What purpose do the statistics serve in the article’s overall message?
  • If an average is cited, is it appropriate or problematic? If the average is not identified, what’s the implication?
  • If statistical comparisons are made — say, between two groups of people — are these comparisons valid? In other words, are similar factors being compared?
  • Are cited statistics given the proper context so you can critically evaluate them? If not, what kind of contextual information do you need?
  • If any correlations are proposed or implied, are they meaningful or spurious? Moreover, look for instances where mere correlation might be mistakenly (or questionably) called causation.
  • Are the data relevant to the issues at hand? In other words, are they substantively significant?
  • Is the presentation of data objective, or are the data being shaped in a discernable way?

Key Resources for Learning More: