Test evidence
These records were written under version 1.1 of the design and use its terms, such as output specification, gate and claim ceiling. The catalogue page maps those terms to the current names.
Result format test
Time of compilation: 11 November 2026, about 13:50 AEDT.
Five techniques were run on the same eight-interview material, once with the catalogue row only (A) and once with the row plus a two to three line output specification (B). A separate agent scored each output against an eight item checklist without knowing which condition produced it. I wrote both the specifications and the checklists, so the scores may favour B. Each condition has one run per technique, so a one point gap is within noise.
Scores (out of 8)
| Technique | A: row only | B: row plus output specification |
|---|---|---|
| TC1-13 Analysis of competing hypotheses | 8 | 8 |
| HE1-14 Framework analysis | 6 | 7 |
| CM1-14 Absence analysis | 6 | 8 |
| TC1-16 Disconfirmation search | 5 | 7 |
| CK-17 Source-span and quotation check | 8 | 7 |
| Total | 33 | 37 |
What the differences were
- Analysis of competing hypotheses: no difference. Both met every item.
- Framework analysis: neither noted which participants gave several quotes. A also altered one quote (Q1 was shortened to 'more advanced', the source reads 'a bit more, uh, advanced').
- Absence analysis: A listed absences in general rather than per hypothesis and did not separate absence in the material from absence in what was collected. B did both.
- Disconfirmation search: A did not state what would count against each hypothesis before searching and did not name the strongest counter evidence. B did both. B left Q6 out of H3 on a reasoned judgement that the checklist counted against it.
- Quotation check: both caught the altered P3 quote, the misattributed P5 quote and the fabricated P9 quote, with no false flags. A stated its string matching method. B did not, and lost one point for that.
Reading the result
The specification helped where the technique has a required structure that a model can skip (absence analysis, disconfirmation search). It made no difference where the technique is already well known (analysis of competing hypotheses) or where the task is mechanical (quotation check). Full method cards were not needed for any of the five. The outputs are in the out folder, and the blind labels are in blind_key.json.
Spot check of 30 rows
Sampled 30 rows across all 12 families: FQ-01, FQ-03, DM-03, DM-12, CM1-04, CM1-12, TC1-08, TC1-19, TC-03, HE1-03, HE1-11, HE-05, PJ-05, PJ-14, PJ-22, AO-02, AO-09, CH-03, CH-11, PE-02, PE-08, MM-03, MM-09, MM-14, IS-01, IS-06, CK-05, CK-12, CK-20, CK-27. Whole-file counts come from scripted checks.
Row problems
- TC1-08, claim_ceiling: "associate". The row's own certainty_limits names that as the ceiling. Suggest "estimate an intervention effect".
- TC-03, access_depth: "requires=aggregate tables". statcheck reads APA-style prose and its fit test rejects tables. Suggest "requires=narrative only".
- HE-05, model_suitability: "model plus code". Its fit test says judging interval assumptions needs a person. Suggest "human judgement required".
- CK-12, model_suitability: "model plus code". It needs human-coded annotations. Suggest "human judgement required".
- CM1-12, source_count: "requires=two or more". The comparator is already external_material_needed, as in siblings CM1-01 and CM1-15, which use "any". Suggest "any".
- HE1-11, consumes: "evidence base:collected". Its related_techniques reason says bias "must be appraised before pooling". Suggest "evidence base:appraised".
- PE-02, consumes and produces: it re-runs "the main result" under several cleaning regimes, yet consumes only the raw corpus and produces no result artefact. Suggest consuming result:draft and producing result:checked.
- CH-03 and CH-11, consumes and produces: both challenge a held judgement but consume "raw dataset or corpus:raw", unlike eight other Challenge rows that consume result, assumptions register or validation report. They also differ from each other (validation report:complete against result:draft) for the same phase. Suggest assumptions register:draft in, validation report:complete out, for both.
- IS-06, linkage: "requires=shared identifier available". Copy detection compares claims and needs no join key (low confidence). Suggest "any".
Patterns
- Frame family, FQ-01 to FQ-05: each consumes charter or question:agreed and produces :draft, so the state falls. FQ-06 does the reverse. The schema rule for question-only techniques forces "agreed", so the rule needs fixing too.
- Frame family, minimum_checks: all six rows share one identical string, including "second technique with different assumptions" at high-stakes. That check suits few framing techniques.
- Input validation: the check library lists it for deterministic techniques only. It appears at every tier in 22 non-deterministic rows (for example CM1-12, PJ-05, MM-03, CK-05). HE-06 is deterministic but uses repeat runs and no input validation.
- access_depth encoding: the field is ordered, so "requires=narrative only" means the same as "any". The file uses "any" in Handle evidence, Integrate and Project forward, and "narrative only" everywhere else (116 rows against 42). Pick one.
- not_for_rules: only 14 of 234 rows use it. HE1-06 and IS-05 use source_dependence, a weight facet, as a hard exclusion.
- Case style: family values are capitalised ("Frame the question"), but the schema says value ids are lower case.
Summary
Of 30 rows checked, 10 had a real problem across 9 findings. Four are clear tag errors (TC1-08, TC-03, HE-05, CK-12). The larger risks are file-wide: the repeated check strings, the input validation misuse and the two encodings of access_depth. No file was edited.