How a score is worked out

Every figure below is read from ledger-strict@0.3.0 as it was released, not written down on this page. A different model is a different set of numbers and a different answer, which is the point of saying which one produced a score. 6 models are released here, and they do not always agree.

The models, and what each one is

A model is a stated set of rules with a version. More than one may be released at a time, and they may disagree; that is the point of naming which one produced a number. Releasing a new version never changes an old one, so a score recorded under an earlier model stays reproducible exactly as it was.

Each name opens that model in full: what it changed, where it differs from the one released beside it, and every number it declares. The numbers shown further down this page are ledger-strict@0.3.0's — show the default instead.

The shape of it

Each piece of evidence linked to a claim is weighed on its own, by multiplying a handful of factors. The strongest weight in each independent group is kept and the rest are set aside. Those kept weights are added to the claim's starting position, and the total is turned into a probability.

relevance × observation × interpretation × authenticity × extraction × provenance × edition = one link's weight

prior as log-odds + the kept weights = posterior log-odds → probability → state

Relevance: how directly the passage bears on the claim

Set by whoever recorded the link, and audited like anything else. A passage that states the claim outright is worth many times one that merely touches the subject, and context-only evidence is worth nothing at all — it is kept because it is worth reading, not because it moves a number.

Observation: what kind of thing the evidence is

A measurement and a rumour are not the same evidence, however relevant each is. This factor is where that difference lives. A model's output weighs nothing — an assistant's opinion is never evidence by itself here, however confident it sounds; it may only point at something a person or a source said, which is then recorded on its own terms.

Interpretation: how many steps of reasoning stand between

A passage that says the thing is worth more than one the reader had to reason from. Each step of interpretation costs 0.15, and the factor never goes below nought:

1 − 0.15 × (number of steps)

So two steps of reasoning on a link is 1 − 0.15 × 2 = 0.70.

Authenticity: whether the passage is really in the source

Whether anyone has checked that the quoted words are actually there, and are really from where they are said to be. Unverified is not penalised — most evidence starts that way and penalising it would punish recording work honestly — but evidence somebody has looked at and doubted is.

Extraction: whether the reading of the passage is faithful

Whether the evidence says what the passage says, as opposed to what somebody hoped it said. Same principle as authenticity: unverified is neutral, and a checked-and-doubtful reading is not.

Provenance: evidence from the claim's own source

When a claim was taken out of a document, a passage from that document shows the claim quotes its source faithfully. It says nothing about whether the source was right. Treating it as support would let any statement corroborate itself simply by being written down.

Edition: which version of the document the claim is about

A living web page is not one document. If a claim says which edition of a source it is about, a reading of a different version of that source weighs nothing against it — an announcement revised two days after publication says something the original did not, and counting the revision against a claim about the original contradicts something the claim never said. The link is kept and shown, marked other edition; it is simply not evidence about this claim.

Independence: the same fact, told twice, counts once

Ten outlets repeating one wire report are one piece of evidence, not ten. Evidence is grouped by where it independently came from, and in each group, for each direction, only the strongest weight is kept. The rest appear in the working at nought, marked dependent strongest only, with the link that was kept named beside them. Nothing is deleted; it is counted once.

Strategy: strongest only per group and direction. Evidence nobody has grouped falls back to a group keyed by its origin and the passage itself, so one page recorded twice as two sources cannot count twice, while two different passages of one document stay distinct.

The prior: where a claim starts before any evidence

A claim with no evidence is not true and not false; it starts at even odds, and the arithmetic is done in log-odds so that evidence adds rather than multiplies. Causal claims start lower, because "A caused B" is a harder thing to establish than "A happened".

Types not listed are not scored at all: they carry no probability and say why instead — a value judgement, a prediction not yet resolved, a rhetorical question. That is a refusal to score, not a score of nought.

Combining: from weights to a probability

The prior becomes log-odds, the kept weights are added to it, and the total becomes a probability:

probability = 1 ÷ (1 + e^{−(prior log-odds + kept weights)})

Rounding is half-even, at 6 places for weights, 4 for the probability and 2 for review coverage. Half-even rather than half-up because it does not drift upwards over many roundings.

The state, and when a number is not shown

A probability becomes a state at these thresholds:

Two states carry no number at all. Insufficient evidence means nothing counted has borne on the claim — not that it is doubtful, that nobody has checked. Not applicable means this model does not score claims of that kind. Both are honest answers, and showing a probability for either would be inventing one.

Stability, and how much the number depends on the settings

The same evidence is scored again with the weights turned down to 0.7 and up to 1.3 of themselves. The spread between those two answers is how much the number is an artefact of the settings rather than of the evidence. A spread within 0.10 is high stability, within 0.25 medium, wider than that low — and fewer than 2 independent groups caps it below high however narrow the spread.

Review coverage: how much of the checking has been done

Always a count of checks, never a percentage of truth. The checks this model declares:

• primary source reviewed
• opposing search done
• independence reviewed
• qualifiers reviewed

A check performed by the author of the claim is recorded and shown, and never raises this figure: coverage means somebody else has looked.

Reading it yourself

Every score carries a trace — the full working, the model, the config hash and the code hash — and the same inputs under the same model reproduce it exactly. The specification sets out the algorithm in full, and a reference implementation in Python reproduces every published example, so this node's answers can be checked without running this node's code.