The Evidence Ladder: How to Hold an Open Question Without Believing Everything

The internet hates a question that has not chosen a side.

A claim appears. Within minutes, one camp has promoted it to revelation. Another has buried it as nonsense. The first side says the pattern is obvious. The second says only an idiot would see a pattern at all.

Both sides want the same thing from you:

Climb faster.

Choose a conclusion before the evidence has finished arriving. Turn uncertainty into identity. Defend the answer as though changing your mind would count as desertion.

But an open question is not a failure of courage.

Sometimes it is the most honest position available.

In the field, uncertainty is not a moral weakness. It is a condition. You do not confuse a contact report with confirmed enemy strength. You do not treat one radio transmission as a complete intelligence picture. You mark what is known, what is likely, what is possible, and what still needs to be verified.

Civilian information warfare deserves the same discipline.

That is what the Evidence Ladder is for.

The False Choice Between Belief and Dismissal

Public arguments often offer only two uniforms:

Believer or debunker.

Believe the theory and you are awake.

Reject it and you are rational.

That framing is convenient for anyone trying to recruit you, because both identities can be manipulated.

The believer can be trained to treat doubt as cowardice.

The debunker can be trained to treat institutional confidence as evidence.

Neither posture guarantees truth.

A question can be reasonable even when the answer remains unknown. A suspicion can deserve investigation without deserving belief. A claim can be false even when the institution denying it has lied before. A source can be biased and still possess relevant evidence. An outsider can be brave and still be wrong.

The point is not to live forever in the middle.

The point is to stop pretending that every claim deserves an immediate verdict.

There is a difference between saying:

“This is true.”

and saying:

“This possibility deserves examination.”

There is also a difference between saying:

“I have not seen sufficient evidence.”

and saying:

“It cannot be true.”

The first statement describes your present evidence.

The second claims knowledge about reality.

Those are not the same thing.

Why Our Minds Climb Too Fast

Human beings do not approach uncertainty as blank machines.

We use mental shortcuts. We notice information that is vivid, familiar, emotionally charged, or similar to a pattern we already recognize. Tversky and Kahneman described several of these shortcuts, including representativeness, availability, and anchoring. They are useful because no one can calculate every possibility from scratch. They also create predictable errors when a compelling story outruns the underlying probabilities.[1]

We are not only shortcut driven. We are motivated.

Ziva Kunda’s influential work on motivated reasoning distinguished between the desire to reach an accurate conclusion and the desire to reach a preferred conclusion. When accuracy is the goal, people can become more careful. When a particular answer protects identity, status, fear, hope, or belonging, reasoning can become a lawyer hired after the verdict.[2]

This does not mean everyone is stupid.

It means intelligence is not a vaccine against self deception.

In fact, a capable mind may be better equipped to build an elegant defense of what it already wants to believe.

That danger exists on every side.

The person who desperately needs the conspiracy to be true can turn every ambiguity into confirmation.

The person who desperately needs the institution to be trustworthy can turn every unanswered question into proof that nothing happened.

The person who has built an audience around certainty may find uncertainty financially inconvenient.

The Evidence Ladder does not remove these pressures.

It gives us somewhere to put them.

The Evidence Ladder

The ladder is a way of classifying what we actually possess before we decide what it means.

From weakest to strongest:

1. Speculation
2. Open question
3. Inference
4. Supported interpretation
5. Documented fact

The project’s evidence order can also be read from strongest to weakest:

Documented fact > supported interpretation > inference > open question > speculation

The order matters because language creates weight.

When speculation is spoken in the tone of fact, the listener receives more certainty than the evidence earned.

When documented fact is dismissed as “just a theory,” the evidence is demoted because it threatens someone’s preferred story.

The ladder is not a ranking of which ideas are socially acceptable.

It is a ranking of how strongly a particular claim is presently supported.

1. Speculation

Speculation is a possibility generated by imagination, analogy, intuition, rumor, or incomplete pattern recognition.

It may be intelligent.

It may even be correct.

But it has not yet earned much weight.

Examples:

• “Perhaps the policy was designed to eliminate smaller competitors.”
• “Maybe the footage was edited to create a false timeline.”
• “It is possible the official had an undisclosed conflict.”
• “The repeated symbol could be deliberate.”

Speculation is not shameful. Every investigation begins before the answer is known.

The mistake is dressing possibility in a uniform it has not earned.

A speculative claim should sound speculative:

• Perhaps
• Could
• Might
• One possibility is
• This would be worth testing

If those words feel too weak, the problem may not be the wording.

The problem may be that the evidence is still weak.

2. Open Question

An open question is more disciplined than speculation.

It identifies a specific gap that available evidence has not resolved.

Examples:

• Who drafted the language that appeared in the final rule?
• Why does the published timeline omit a documented meeting?
• Were the researchers aware of the sponsor’s requested outcome?
• What evidence should exist if the allegation is true?
• What evidence should exist if it is false?

An open question does not smuggle the answer into the question.

Compare:

“Why did the agency cover this up?”

with:

“What explains the difference between the internal record and the public statement?”

The first assumes concealment.

The second identifies a discrepancy.

That distinction may feel less satisfying. It is also far more difficult to dismiss.

A clean question is a sharp instrument.

A loaded question is a conclusion hiding behind punctuation.

3. Inference

An inference is a conclusion drawn from evidence, but not directly stated by the evidence itself.

Suppose meeting records show that a regulator met repeatedly with one industry group before issuing a rule favorable to that group.

The meetings are documented facts.

The favorable rule is a documented fact.

The claim that those meetings caused the rule is an inference.

It may be a strong inference. It may be the best explanation. But the documents do not automatically tell us what occurred in every conversation or what would have happened without them.

Inference is where much serious analysis lives.

It is also where people most often forget to label their work.

The responsible form sounds like:

“These records support the inference that the industry had unusual access, but they do not by themselves establish that the outcome was purchased.”

That sentence does not weaken the finding.

It tells the reader exactly where the evidence ends and the reasoning begins.

4. Supported Interpretation

A supported interpretation is an explanation backed by multiple relevant facts, independent corroboration, a plausible mechanism, and meaningful resistance to competing explanations.

This is stronger than a single inference because the interpretation survives contact with other possibilities.

A supported interpretation might include:

• Primary records
• Independent sources
• A documented timeline
• A mechanism connecting cause and outcome
• Evidence against obvious alternatives
• Consistency across different kinds of data

The word supported matters.

It does not mean immortal.

A well supported interpretation can still change when better records emerge, methods improve, or an overlooked explanation fits the evidence more completely.

Science operates through this kind of accumulated testing. The National Academies distinguishes reproducibility, obtaining consistent computational results from the same data and methods, from replicability, obtaining consistent findings across studies using new data. Neither process turns scientists into infallible authorities. Both help determine whether a result survives beyond one analysis or one research team.[3]

A single paper can be evidence.

It is rarely the entire ladder.

5. Documented Fact

A documented fact is the strongest category on the ladder.

It is something directly established by reliable records, physical evidence, verified data, authenticated media, sworn testimony, or other evidence appropriate to the claim.

Examples:

• A contract was signed.
• A payment was made.
• A meeting occurred.
• A law contains specific language.
• An email was sent from one verified account to another.
• A laboratory produced a particular measured result under stated conditions.

But even documented facts have limits.

A contract proves that an agreement existed.

It may not prove every motive behind it.

A meeting record proves that people met.

It may not prove what they intended.

A photograph proves that something appeared within a frame at a particular moment, assuming authenticity and context are established.

It may not explain what happened outside the frame.

Facts are the strongest rungs. They are not automatic interpretations of themselves.

Facts Do Not Assemble Their Own Story

This is where disciplined people still get trapped.

We collect several true things and assume the story connecting them must also be true.

But the same set of facts may support more than one explanation.

A public official receives campaign support from an industry.

The official later votes for a policy that benefits that industry.

Both facts may be documented.

Possible explanations include:

• The support purchased influence.
• The industry supported the official because they already shared the same policy position.
• The official believed the policy served both the public and the industry.
• Staff analysis, constituent pressure, party discipline, or unrelated events shaped the vote.
• Several of these forces operated together.

The facts narrow the field.

They do not always identify the only survivor.

Research on conspiracy theories emphasizes that allegations of secret coordination may, in principle, turn out to be true. The category does not make a claim false by definition. But explanations involving hidden coordination still need descriptive clarity and evidence appropriate to the alleged mechanism.[4]

The goal is not to protect official narratives from suspicion.

It is to protect the facts from being drafted into a story they did not volunteer to tell.

Absence of Evidence: Ask What Should Exist

“A lack of evidence does not prove it is false.”

Correct.

But incomplete.

The next question is:

If the claim were true, how likely is it that we would have found evidence by now?

Research on reasoning from absent data shows that missing evidence becomes informative when that evidence would reasonably be expected to appear under the hypothesis being tested.[5]

Imagine two alleged operations.

The first could be completed by five people, inside a classified program, using verbal instructions and records scheduled for destruction.

The second would require fifty thousand equally informed participants, large budgets across several agencies, visible construction, years of perfect coordination, no leaks, no defectors, no contradictory decisions, and no physical traces.

Public silence means different things in those two cases.

Absence is not automatically proof of innocence.

It is not automatically proof of concealment either.

Ask:

• What traces should the claim produce?
• How durable would those traces be?
• Who would have access to them?
• Would independent institutions hold copies?
• Has anyone searched where the evidence should appear?
• Does the claim predict anything before it happens?
• What observation would make the claim less likely?

A theory that explains every missing record as proof of perfect concealment becomes difficult to test.

A theory that cannot lose can never truly win.

The Source Is Part of the Evidence

A screenshot may accurately reproduce a sentence and still conceal the paragraph that changes its meaning.

A document may be authentic and selectively presented.

A credentialed expert may understand the subject and have an undisclosed conflict.

An anonymous account may lack credentials and still direct you to a genuine primary record.

Source evaluation is not the practice of deciding which tribe is allowed to speak.

It is the practice of asking what each source can actually establish.

The Digital Inquiry Group teaches lateral reading, a method that begins by leaving an unfamiliar page and investigating who is behind it, what other credible sources say, and how the claim appears elsewhere.[6]

Mike Caulfield’s SIFT method offers a memorable sequence:

• Stop
• Investigate the source
• Find better coverage
• Trace claims, quotations, and media to their original context[7]

This is not a demand to trust whatever appears first in a major search engine.

It is a demand to stop letting one source grade its own homework.

When tracing a claim, look for:

• The earliest available version
• The complete document
• The full video rather than the clip
• The original data rather than the graphic
• The exact wording of the law
• The methodology behind the statistic
• The funding and conflicts disclosed by the authors
• Independent reporting that did not originate from the same press release

The source is not the conclusion.

It is part of the chain of custody.

Corroboration Is Not Repetition

Ten articles can repeat the same claim and still represent one source.

A hundred social posts may all trace back to one anonymous screenshot.

Three television networks may rely on the same wire report.

Several research summaries may all cite the same small study.

Repetition creates familiarity.

Familiarity can feel like confirmation.

It is not the same thing.

Independent corroboration means separate evidence streams converge without merely copying one another.

A claim becomes stronger when:

• Different witnesses provide compatible accounts without coordinating them
• Financial records support a timeline established by communications
• Physical evidence matches documented movement
• Separate research teams obtain compatible findings with new data
• Adversarial sources agree on a narrow factual point

Corroboration is especially valuable when the sources have different incentives.

Two allies agreeing may be meaningful.

Two opponents agreeing on a fact may be harder to ignore.

Confidence Is a Dial, Not a Badge

Most people speak in three settings:

• True
• False
• I have not looked

Reality deserves a better control panel.

Confidence can be expressed as:

• Very low
• Low
• Moderate
• High
• Very high

Or, when useful, as a probability range.

The purpose is not to cosplay as a statistician.

The purpose is to stop pretending that every belief has equal strength.

Good calibration means that confidence roughly matches accuracy over time. In geopolitical forecasting research, probability training, team discussion, and performance tracking improved both calibration and predictive accuracy.[8]

That lesson applies beyond prediction.

Record what you believe.

Record how confident you are.

Record why.

Record what would change your mind.

Then return later and see whether your certainty was earned.

Memory is an excellent public relations department. It quietly edits old confidence after the outcome is known.

Written forecasts leave fewer places to hide.

A disciplined thinker does not say:

“I was basically right.”

They ask:

“What did I predict, at what confidence, and what actually happened?”

The Strongest Counterargument: We Still Have to Act

There is a danger in worshiping uncertainty.

A person can demand endless evidence because they do not want to make a decision. Institutions can use “more research is needed” as a delay tactic. Bad actors can manufacture doubt long after the balance of evidence is strong enough to justify action.

The Evidence Ladder must not become a staircase to nowhere.

We routinely act without certainty.

A commander acts with incomplete intelligence.

A doctor treats before every possible diagnosis is eliminated.

A jury reaches a verdict without omniscience.

A family evacuates before the fire reaches the porch.

The evidence required to believe a claim with high confidence is not always the same as the evidence required to take a proportionate precaution.

You may support an audit because the discrepancy is serious, even if wrongdoing is not proven.

You may pause a product because early safety signals are concerning, even if causation remains uncertain.

You may preserve records because a claim deserves investigation, even if it remains an open question.

Action should match both the possible harm and the strength of the evidence.

The answer to uncertainty is not paralysis.

It is proportional response.

A Five Step Field Drill for Holding the Question

When a claim matters, use this sequence.

1. State the claim

Write one sentence that can be tested.

Not:

“Everything is connected.”

Instead:

“The final policy language was drafted primarily by representatives of the industry it regulates.”

Remove emotional adjectives.

Separate what happened from what you think caused it.

2. Find the original source

Locate the earliest available document, recording, dataset, testimony, or direct statement.

Ask:

• Is it complete?
• Is it authentic?
• Is it current?
• What does it actually say?
• What does it not say?

3. Check corroboration

Look for independent evidence.

Do not count copies as confirmations.

Search for records created for different purposes, especially records that would exist even if nobody were trying to prove your theory.

4. Test alternatives

Write the strongest competing explanation.

Not the dumbest one.

Ask what evidence each explanation predicts.

A strong theory should explain more of the record with fewer unsupported assumptions.

5. Assign confidence

Choose a level:

• Speculation
• Open question
• Inference
• Supported interpretation
• Documented fact

Then assign your confidence within that level.

An inference can be high confidence.

A documented fact can be low confidence if the document’s authenticity is uncertain.

Finally, write:

“I would change my mind if…”

If you cannot complete that sentence, you may not be investigating anymore.

You may be protecting an identity.

Open Questions Are Not Empty

There is a tendency to treat “we do not know” as though it means “nothing is there.”

That is not what it means.

An open question can be morally important.

It can justify records requests, audits, experiments, hearings, journalism, whistleblower protections, or further research.

The National Academies’ 2025 report on science misinformation describes a complex information ecosystem in which inaccurate claims can arise intentionally or inadvertently from many sources. It recommends a systems view rather than reducing the entire problem to a few malicious actors.[9]

That same humility should apply to investigation.

Some false claims are manufactured.

Some are misunderstandings.

Some emerge from institutional opacity.

Some begin as reasonable questions and harden into identities.

Some official statements are accurate.

Some are technically accurate while omitting the part the public most needed to know.

An open question preserves the possibility of discovery without pretending discovery has already occurred.

The Discipline of Changing Your Mind

Changing your mind is often treated as evidence that you were weak, captured, fooled, or never serious.

That is childish.

Refusing to update when the evidence changes is not strength.

It is surrender to a previous version of yourself.

The stronger posture is:

“This was my conclusion based on the evidence available then. The evidence changed. So did my conclusion.”

That is not inconsistency.

That is calibration.

The real test of a research method is not whether it always confirms what you suspected.

It is whether it can also tell you that the pattern was weaker than it looked.

Sometimes you will climb the ladder.

Sometimes you will move down.

Sometimes the most honest result of months of work is:

“The allegation remains possible, but I found less support than I expected.”

That sentence will not go viral.

It may save your integrity.

The Courage to Stay at “I Don’t Know Yet”

“I do not know yet” is not a permanent address.

It is a checkpoint.

It tells us where the evidence currently stops.

The culture rewards certainty because certainty is easy to sell. It makes strong thumbnails, clean slogans, loyal audiences, and simple enemies.

But reality is often under no obligation to become marketable.

The mature investigator can carry an unresolved question without stuffing the empty space with fear.

They can remain suspicious without becoming consumed.

They can recognize a pattern without kneeling before it.

They can challenge institutional authority without giving automatic authority to whoever opposes it.

They can pray, study, test, listen, and wait.

That waiting is not passivity.

It is fire discipline.

Do not shoot at movement because you wanted a target.

Identify what you are looking at.

Pull Another Thread

Who Benefits? How Incentives Shape the Stories We Are Told
Follow the supply lines of money, access, algorithms, and audience demand without mistaking motive for proof.

Conspiracy, Capture, or Coincidence? How Power Moves Without a Secret Room
Examine how power can move through contracts, regulations, professional networks, and ordinary institutional incentives.

Not Every Question Is a Conspiracy
Build a grounded standard for asking difficult questions without treating suspicion, possibility, or pattern as evidence.

Thread Note

A question is not proof.

A possibility is not a probability.

A pattern is not automatically a plan.

A document is not automatically the story surrounding it.

And “unproven” does not mean “false.”

Climb slowly.

Name the rung.

Trace the source.

Test the strongest alternative.

Let confidence rise only when the evidence does.

The goal is not to become certain first.

The goal is to become honest enough to know what you know, what you infer, what you suspect, and what remains hidden in the dark.

Hold the question.

Follow the evidence.

Keep your hands clean.

Research Notes & Further Reading

[1] Amos Tversky and Daniel Kahneman. “Judgment under Uncertainty: Heuristics and Biases.” Science, 1974. Foundational research on representativeness, availability, anchoring, and predictable errors in judgment under uncertainty.
https://doi.org/10.1126/science.185.4157.1124

[2] Ziva Kunda. “The Case for Motivated Reasoning.” Psychological Bulletin, 1990. Distinguishes accuracy goals from directional goals and explains how desired conclusions can influence the information and reasoning strategies people use.
https://doi.org/10.1037/0033-2909.108.3.480

[3] National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science, 2019. Defines reproducibility and replicability and explains their role in testing whether findings survive beyond one analysis or study.
https://doi.org/10.17226/25303

[4] Karen M. Douglas and Robbie M. Sutton. “What Are Conspiracy Theories? A Definitional Approach to Their Correlates, Consequences, and Communication.” Annual Review of Psychology, 2023. Reviews how conspiracy claims are defined and why the category does not, by itself, determine truth or falsity.
https://doi.org/10.1146/annurev-psych-032420-031329

[5] Anne S. Hsu, Andy Horng, Thomas L. Griffiths, and Nick Chater. “When Absence of Evidence Is Evidence of Absence: Rational Inferences From Absent Data.” Cognitive Science, 2017. Explains why missing evidence becomes informative when that evidence would be expected under the hypothesis being tested.
https://doi.org/10.1111/cogs.12356

[6] Digital Inquiry Group. “Intro to Lateral Reading.” Civic Online Reasoning curriculum. Introduces the practice of leaving an unfamiliar page to investigate who is behind it and what other credible sources say.
https://cor.inquirygroup.org/curriculum/lessons/intro-to-lateral-reading/

[7] Mike Caulfield. “SIFT: The Four Moves.” 2019. A practical framework for stopping, investigating the source, finding better coverage, and tracing claims to their original context.
https://hapgood.us/2019/06/19/sift-the-four-moves/

[8] Barbara Mellers et al. “Psychological Strategies for Winning a Geopolitical Forecasting Tournament.” Psychological Science, 2014. Reports that probability training, team collaboration, and performance tracking improved calibration and forecasting accuracy.
https://doi.org/10.1177/0956797614524255

[9] National Academies of Sciences, Engineering, and Medicine. Understanding and Addressing Misinformation About Science, 2025. Uses a systems approach to explain how science misinformation can arise intentionally or inadvertently and recommends improving access to high quality information.
https://doi.org/10.17226/27894