You test something to rule out a cause, it passes, and you move on — but the test never touched the part that was broken. Here is how a comparison goes wrong, and the four questions that make one worth running.

The report says zero. You suspect the report is broken, so you check it against a night you remember being busy.

It shows the right figure. Good — the report works, so the zero is real.

⚠️ Except you checked the wrong half. The report reads from the till; the zero was in the online bookings. Your test never went near the part that failed, and it passed anyway.

The short version

A comparison only proves things about the path it actually travelled
Testing from halfway along never sees the part before it
The thing you compare against can be empty itself — then it proves nothing
Sometimes there is nothing to compare against. Write that down; it is a result
An explanation that fits is not the same as an explanation that is correct

Why a passing test is not reassurance

What you did	What it actually shows
Checked the report against a known night	The report can display a figure
Checked one till drawer	That drawer was counted
Checked the booking system	Bookings that reached the system arrived

⚠️ None of those covers the phone call nobody entered. The test passed and the gap is still there.

One: start the test where the break could be, not where it is easy

The most common error is to begin the comparison halfway along.

The path   enquiry → someone answers → entered → appears on report
Your test  entered → appears on report
⇒ everything before "entered" is untested

⚠️ The easy half to test is almost always the half that works, because it is the visible half.

The question that fixes it

Where could this break? Does my check pass through that point?

⇒ If the answer is no, the check is not wrong — it is just about something else.

Two: the thing you compare against can be empty

We ran into this exactly. A new folder was showing zero files, so we checked our method against a different folder that should have had plenty.

New folder      0 files
Comparison      0 files  ← also empty
⇒ proves nothing

⚠️ We had chosen a comparison that was empty for its own reasons. Only on the third attempt — a folder we knew was full — did the method prove itself, and the original zero turn out to be genuine.

The rule

Pick something you are certain has content.
If your comparison also comes back empty, it is not a comparison.

⇒ In a venue: if you are testing whether your takings report works, check it against last Saturday, not against a Tuesday in January.

Three: sometimes there is nothing to compare against

There is no rule saying a comparison must exist.

One booking system, one till, one process
⇒ nothing to hold it up against

⚠️ Do not invent one. Creating a fake booking to test the path puts a fake booking in your records, and it will outlive the test.

⇒ Write down "no comparison available" and say so when you report. That is an honest result, and it tells the next person what was and was not established.

⇒ On records that quietly become permanent: the note that was true when you wrote it.

Four: an explanation that fits is not an explanation that is right

This is the subtler one.

Something fails. You spot a plausible cause. You stop looking.
⇒ the cause you spotted may be real and still not be the reason

⚠️ We did this twice in one morning. A batch of links returned nothing; a difference in how they were written looked like the cause. It was a real difference — and it was not why they failed. The actual reason was an invisible character at the end of each line.

The test of an explanation

Does it fit?          ← necessary
Does fixing it fix the problem?  ← the actual test

⇒ An explanation you have not acted on is a guess with detail attached. Try the fix. If the problem persists, the explanation was decoration.

Two ways an explanation goes wrong

Wrong direction  the cause is somewhere else entirely
Right direction, too narrow  the cause is real but only part of it

⚠️ These need opposite responses. Too narrow, you add to it. Wrong direction, adding to it makes it worse.

Five: the venue version of all four

Take a real case: a regular says he never got his confirmation.

Bad check   send yourself a test message ⇒ it arrives ⇒ "system works"
             (tests your own address, not his)
Better      look up his record, check the address on it, read it aloud to him
             (tests the part that could actually be wrong)

⚠️ The bad check is faster, feels rigorous, and proves nothing about his booking.

The same shape elsewhere

"The card machine works"     - tested with your own card, not the one that failed
"The rota is correct"        - checked the week you remember, not the week in dispute
"Prices are right"           - checked the ones you set, not the ones somebody changed

⇒ In each case the test travelled a path that was never in doubt.

Six: when somebody hands you a reason

Colleagues, suppliers and systems all hand you explanations. Treat a received explanation the same as your own.

"It's because the system was down"
"It's because that guest always books late"
"It's because of the bank holiday"

⚠️ Each may be true and still not be the cause of this specific case. The question is the same: does it fit this one, and does acting on it fix this one.

⇒ Useful phrasing that avoids an argument: "Is that from looking, or is that the likely answer?" Both are valuable; they are just different things.

⇒ Related: who did it is not what they did.

Seven: a number that contradicts your explanation

The strongest signal that an explanation is too narrow is a second number that will not sit with it.

Explanation  "nothing was sent, so of course nothing arrived"
Second number "four records were received"
⇒ both cannot be true as stated

⚠️ When two of your own numbers disagree, stop. One of them is measuring something you have not named yet. In our case the four were not messages at all — they were created by the setup itself.

The habit

Write the explanation down. Then look at every other number
you have from the same morning and ask if any of them argue with it.

⇒ Two minutes, and it is where most wrong conclusions get caught.

Numbers worth keeping

One   For each check you run: which part of the path does it cover?
Two   What did you compare against, and was it known to have content?
Three For each explanation: have you tested the fix, or only the fit?
Four  Any number from the same day that contradicts it?

⚠️ Two is where most checks fall over, and it takes one sentence to record.

The card for the office

────────────────────────
  Before trusting a check

  1. Does it pass through the part that could be broken?
  2. Is the thing I compared against known to have content?
  3. If there is nothing to compare against - say so.
  4. Did I test the fix, or only that the reason sounded right?

  A test proves things about the path it travelled.
────────────────────────

Common objections

"The test passed, so it's fine"

⚠️ It is fine along the path the test took. Name that path out loud and the gap usually appears.

"We don't have time to test properly"

⇒ The four questions take a minute. Re-running a check that proved the wrong thing takes an afternoon.

"It's obviously the reason"

⚠️ Then the fix will work, and that takes five minutes to confirm. Obvious and correct are different.

"There's nothing to compare it to"

⇒ Then write that down. "No comparison available" is a finding, and it stops the next person assuming one was done.

"Our supplier already looked into it"

⇒ Ask which part they checked. Suppliers, reasonably, test their own half.

What to do this week

One   Take one check you rely on. Name the path it covers.
Two   Find the part that could break. Is it on that path?
Three Next time you compare, pick something you know has content.
Four  For your next explanation, test the fix, not the fit.
Five  Look for a number from the same day that argues with it.

Summary

A check proves things about the path it travelled, nothing more
Starting halfway along leaves everything before it untested
A comparison that is empty itself proves nothing
"No comparison available" is an honest, useful result
Fitting is not the same as being right — test the fix
Two of your own numbers disagreeing means one measures something unnamed

A check that passes tells you where you looked. Whether that is where the problem lives is a separate question, and only you can ask it.

The Check That Proved The Wrong Thing