From Data to Decision
Before a conclusion is allowed to drive a decision, it must pass a fixed checklist covering how the data was collected, how it was summarized, and how it was displayed. · 13 min
This is the last folio, and it collects the whole course into one habit. Every claim you meet — in an advertisement, an article, a meeting — arrives as a finished conclusion, with the reasoning already thrown away. Your job is to rebuild the reasoning and test it before you act. The good news is that the tests are always the same, and they run in a fixed order. Learn the checklist once and you can apply it to any number anyone ever shows you.
Guess before you learn
A post claims a supplement 'cut sick days by 40%.' Of the following, which question should you ask first?
Collection comes first because nothing downstream can repair it: a beautiful chart and a clever average built on a biased or uncontrolled sample are still worthless. The other questions matter, but only after you know the numbers were gathered in a way that could support the claim at all.
9–12
3–5
To judge a claim, run a short checklist. First, how was the data collected — who got asked, and were they picked fairly? Second, how was it summarized — is the average an honest middle? Third, how is it shown — does the chart play fair with its axis? Only if all three pass should the claim change what you do.
6–8
A defensible conclusion has to survive a checklist with three stages: collection, summary, and display. Collection asks how the data was gathered and whether the sample was fair. Summary asks whether the numbers chosen — the average, the spread — honestly describe the data. Display asks whether the chart distorts what it draws.
You apply the stages in order, and any one of them can sink the claim. A perfect chart of badly collected data is still worthless; a fair sample summarized by a misleading average is still misleading. The conclusion is only as strong as its weakest stage — so you check every stage before you let the conclusion drive a decision.
9–12
The checklist is the whole course, folded into four questions. Collection: who was sampled or assigned, and how — the material of sampling and study design. Summary: is the center and spread honest for this distribution's shape? Display: does the chart's axis and time window tell the truth? Conclusion: is this an association or a genuine cause?
Applying them in order matters, because each stage assumes the one before it held. There is no point debating whether an average is fair if the sample it averages was biased from the start. A conclusion inherits the weakest stage in its chain: fix everything except collection and a biased study still proves nothing. This ordered scepticism is what turns raw data into a decision you can defend.
K–2
Before you believe a surprising number, ask three things. Where did the numbers come from? What do they leave out? Does the picture play fair? If any answer is bad, wait before you decide.
Undergrad
Treat a claim as a chain of inferences running from a target population to an action: sampling frame, measurement, estimator, visual encoding, and causal warrant. Each link is a separate, checkable assumption, and the argument is valid only if every link holds. The checklist is a lightweight adversarial review — a way of asking, at each joint, what would have to be true for this step to be sound.
Provenance is central: undisclosed choices in how a sample was drawn, how outliers were handled, or which time window was plotted are exactly where sound-looking claims fail. Pre-registration and open data exist to make these joints inspectable in advance. The discipline is not cynicism but calibration — apportioning your confidence to the weakest documented link rather than to the polish of the presentation.
Postgrad
Formally the pipeline from data to decision is a composition of inferential steps, each carrying an assumption that is in principle falsifiable: representativeness of the sampling design, validity of the measurement model, unbiasedness of the estimator, faithfulness of the graphical encoding, and identifiability of the causal estimand. Error propagates multiplicatively; overall credibility is bounded by the least defensible stage, not the average.
The replication crisis is best read as systematic failure at these joints — flexible stopping rules and specification search inflating false positives, selective reporting corrupting the published record. A structured checklist functions as a cheap screening filter applied before any Bayesian update, and, in decision-theoretic terms, as a guard on the expected loss of acting: the cost of a confident wrong conclusion is what the whole procedure is built to bound.
the checklist
A fixed sequence of questions — collection, summary, display, conclusion — applied in order to any statistical claim. A claim is defensible only if every stage holds; the weakest stage sets the ceiling on your confidence.
Watch the checklist run on a real claim. A viral post announces: Students who used our study app scored 20% higher. Read it stage by stage, and notice that you do not need any new statistics — only the questions you already own. The point of the checklist is that it is mechanical: you ask the same things in the same order, and the claim's weaknesses surface on their own.
Run the checklist: 'Students who used our study app scored 20% higher' — the steps fade as you master them
They chose to use it themselves; there is no comparison group set up in advance
Higher than the students who did not use it, on an unstated baseline
The post shows a headline figure and no chart to inspect
Observational and self-selected, so it shows association, not that the app caused the gain
Practice — new ink and old, interleaved
1.An advertisement says '9 out of 10 dentists recommend our brand,' based on a survey the company ran but will not describe. Which stage of the checklist is most clearly in doubt?
2.Cities that hire more police tend to report more crime. Before concluding police cause crime, the checklist's conclusion stage asks you to consider:
3.The same right-skewed wait times have a mean of 22 minutes and a median of 14. Which is the more honest headline figure for a typical wait?
4.Without looking back: what is the one question that decides whether a study may claim cause, and what answer is required?
Did the researchers assign the treatment, ideally at random? Only a study that did — a randomized experiment — may claim cause; a study that merely observed can claim association only.
How close were you? Grade yourself honestly — it sets your review date.
5.Match each correlation to the kind of explanation most likely behind it.
6.A poll reports 54% support with a margin of error of ±3 points at 95% confidence. Which reading is correct?
7.A bar chart of two candidates' support has its vertical axis running from 44% to 48%. What does this most likely do?
8.Which sample of a school's students is least likely to be biased?
9.Which finding may fairly be reported as evidence that a treatment CAUSES an improvement?
10.Without looking back: name the four stages of the checklist in order, and state the rule about which stage decides your overall confidence.
Collection, summary, display, conclusion — applied in that order; your confidence is capped by the weakest stage, so every stage must hold for the conclusion to be defensible.
How close were you? Grade yourself honestly — it sets your review date.