CK-27 Adversarial testing of an extraction pipeline (red-teaming; chaos testing)
Probe an extraction pipeline with hostile or odd inputs such as instructions inserted in source text, contradictory documents and missing data, and produce a list of failures found.
What it does
Probes the pipeline with hostile or odd inputs, such as instructions inserted in source text, contradictory documents and missing data. Artefact: a list of failures found. Read the failures that change a reported result.
When to use it
Use when a pipeline reads documents from outside sources and you need to know whether planted or odd content can change its output.
When it does not fit
Fits pipelines that handle untrusted documents. Misfit: the testers also built the system. Both sources aim at safety and security more than analytic accuracy (inference).
Result format
The result is a list of the failures found, each with the hostile or odd input that caused it, such as inserted instructions, contradictory documents or missing data. It marks which failures change a reported result.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | a runnable pipeline and a set of test inputs written by someone other than its builder |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the testers also built the system. Both guidance sources aim at safety and security more than analytic accuracy. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | risk |
| Question answered | what next |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
| high-stakes | expert review | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Follows instructions embedded in source documents. |
| Error sources | input data; language model; expert opinion |
| Certainty limits | Certainty drops when testers also built the pipeline (risk of bias) and when test inputs cover only known attack types (indirectness). |
Related techniques
- should follow PE-10 Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents): structured extraction is exposed to inserted instructions
- should follow PE-13 Retrieval-augmented analysis (RAG (retrieval-augmented generation); grounded question answering over a corpus): retrieved passages can carry inserted instructions
- pairs with CK-25 Error taxonomy and error analysis (failure mode analysis; confusion analysis): failures found are classified by type
Evidence and tag basis
Documented. NIST AI 600-1 calls for red-teaming, and the Australian guardrail 4 includes it for general-purpose AI. (NIST AI 600-1; Voluntary AI Safety Standard, 10 guardrails)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.