What happened to this number before it reached the slide?

A number usually arrives at the end of a long pipeline. It shows up as a percentage on a dashboard, a bar on a chart, a model output with a confident figure attached. By the time it reaches the screen, every choice that decides whether it means anything has already been made upstream: who was counted, what the baseline was, which cuts survived the edit.

The person reading it sees only the last step. The arithmetic is visible, and the decisions behind it are hidden. That asymmetry explains how careful people can nod at a chart they privately distrust, and how two honest teams can look at the same data and reach opposite conclusions.

Working analysts handle this with a habit more than a technique. The interrogation precedes the arithmetic. Before a number is allowed to persuade, it gets walked backwards up its own pipeline with a short list of questions. The questions need no formulas and no software. They work with a slide on the screen and nothing in hand, and each one opens a different stretch of the pipeline to inspection.

Start from the decision

The first question points at the request itself: what will be done differently depending on the answer? Tracing a request backward to the decision it serves, a practice called decision-back framing, reveals what kind of analysis the situation calls for. Most decisions need a comparison, one option measured against another in a way that can favor or disfavor an action. Many requests arrive asking for a description: a single figure, a distribution, a total.

A decision-shaped question also defines when the work is finished, because its answer determines what happens next. The companion discipline is to name, before seeing any data, the result that would change the conclusion. Declaring the outcome measure and the threshold in advance is what makes an investigation capable of informing a choice. When a request specifies the exact cut in fine detail and shows little interest in which comparison could move the outcome, it is usually a request for ammunition, and recognizing that early saves everyone time.

Every rate makes a claim about who counts

Once the question is settled, attention moves to the fraction. Every rate divides events by the units that were at risk of producing them, and that bottom line, the denominator, carries a quiet claim about who belonged in the count. Many disputes that look like arguments about a statistic turn out to be arguments about two different denominators.

Changing the denominator, without touching a single count, can reverse a conclusion entirely. A case fatality rate divides deaths by confirmed cases, so its value moves with testing capacity as much as with the disease. A dramatic trend can appear in the very period a team changes how the eligible population is identified, with no change at all in underlying behavior. The useful check is to ask for the opportunity base, the events that were genuinely at risk of producing the outcome, and whether its definition held steady across the whole series.

The people a pattern cannot see

The denominator question has a sharper cousin: who is absent from the data altogether? Every record passed some entry condition to appear. The customer had to answer the survey, the candidate had to clear the screen, the project had to be written up. Those filters rarely act at random. They tend to correlate with the very outcome under study, so the cases most informative about a problem are often the ones the dataset quietly dropped.

Asking who the pattern cannot see comes before reading the pattern. Abraham Wald's analysis of returning bombers turned on the aircraft missing from the sample, and the same logic applies to churned customers who skipped the exit survey and to failed projects that left no record. Provenance belongs to the same question. Much organizational data is residue from a workflow built for another purpose, such as billing, compliance, or keeping a sales pipeline tidy. A loss reason chosen in four seconds to clear a CRM record describes the recording system as much as the customer. Data answers the question its process was built for, and an analyst's first job is to learn what that question was.

Four explanations for one relationship

With the inputs examined, the next temptation arrives: two things moving together. An observed relationship has four live explanations. One variable may cause the other, the second may cause the first, a third factor may drive both, or the pairing may be coincidence. A correlation is a hypothesis, and a scatter plot licenses holding all four explanations open at once.

Evidence narrows the field one candidate at a time. A dose-response pattern, a plausible mechanism and a clear order in time each remove some explanations. The sharpest tool is the counterfactual: what would have happened to the same units if the change had been absent? That outcome is unobservable for any single case, which is why randomization carries so much weight. Assigning treatment by chance balances the hidden third variables on average. Where an experiment is impossible, natural experiments and comparisons of trends before and after a change can approximate the same logic, provided their assumptions are stated openly.

The width of the interval is a finding

Even a well-identified effect comes with a range. A point estimate is that range with the information about precision stripped away. The range itself is a statement of precision: it says how much the evidence can settle. A wide interval is a finding about the limits of the data, and reporting it well means giving the range, the likely direction, and what additional data would narrow it.

The same honesty applies to judgment over time. Calibration asks whether events forecast at 70 percent happen about 70 percent of the time. Keeping score of stated probabilities turns vague confidence into a skill that improves with feedback, and it gives an analyst a record of when past certainty was earned.

When measures meet incentives

The last stretch of the pipeline is the organization that receives the number. Goodhart's Law describes what happens when a measure becomes a target: people respond rationally to the incentive, and the measure drifts away from the thing it once tracked. The drift is a predictable response to design, and design is where the defense lies. Pairing each target with a counter-metric that watches what the target might erode keeps the original signal readable.

The strongest protection is an audit trail. When an analyst knows a colleague must be able to retrace every step, the choices change while the work is still underway: exclusions get recorded, alternative cuts get reported beside the one that worked, assumptions get written down. Reproducibility is a property of the analysis itself. It makes a correct result more likely, as well as checkable.

Questions asked early

Taken together, these questions form a single habit of mind. What decision is this for, and what result would change it? What is the denominator, and who is absent from the data? Which of the four explanations does the evidence support? How wide is the range? What will this measure become once people are rewarded for moving it, and could someone else retrace the path that produced it?

Each one can be asked in a meeting with a chart on screen and nothing in hand. The arithmetic is almost always the easy part. The interrogation is where a finding earns the right to hold, and it begins with the history of a number, well before anyone reads its value.