startthat.work home

HE1-01 Admiralty system (source reliability and information credibility grading)

Assigns each report a two-character grade for source reliability (A to F) and information credibility (1 to 6) and produces an appraised evidence base.

Handle evidenceEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Gives each report a two-character grade, rating the source (A to F) and the information (1 to 6) separately. Artefact: a code such as B2; read the letter as the source's history and the number as how well the item is corroborated.

When to use it

Use when you need to say how far to trust each report from a source with a known track record, and whether other independent reports back it.

When it does not fit

The source is new, anonymous or has no comparators (default F and 6). The grades are ordinal, and reliability from past performance does not predict a new claim.

Result format

The result is a table with one row per report, giving a two-character grade such as B2, where the letter (A to F) rates the source and the number (1 to 6) rates the information. It states what each grade rests on, the source's history for the letter and the corroboration for the number.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=source; requires=claim
Producer interestChanges the orderprefers=sponsored; prefers=self-interested seller; prefers=unknown
Source dependenceChanges the orderprefers=independent; prefers=unknown
OriginChanges the orderprefers=secondary; prefers=tertiary; prefers=unknown
Appraises source credibilityApplies in some casesyes

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)The source is new or anonymous with no comparators, so the default grade F6 carries no information. Do not read a grade from past performance as a prediction about a new claim.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsgather
Kind of findinggap; contradiction
Question answeredwhat; who
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureAssigns a letter grade from the tone of the report instead of the source's record, or scores information credibility without checking whether the corroborating reports are independent.
Error sourcesinput data; coder or rater judgement; source selection
Certainty limitsCertainty drops when the track record is thin, when corroborating reports copy each other (inconsistency and risk of bias) and when grades are read as more than ordinal.

Evidence and tag basis

Documented. Read on a secondary page citing UK and US military publications; the primary document was not opened. (Wikipedia, Admiralty code)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.