startthat.work home

CK-27 Adversarial testing of an extraction pipeline (red-teaming; chaos testing)

Probe an extraction pipeline with hostile or odd inputs such as instructions inserted in source text, contradictory documents and missing data, and produce a list of failures found.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: describe

What it does

Probes the pipeline with hostile or odd inputs, such as instructions inserted in source text, contradictory documents and missing data. Artefact: a list of failures found. Read the failures that change a reported result.

When to use it

Use when a pipeline reads documents from outside sources and you need to know whether planted or odd content can change its output.

When it does not fit

Fits pipelines that handle untrusted documents. Misfit: the testers also built the system. Both sources aim at safety and security more than analytic accuracy (inference).

Result format

The result is a list of the failures found, each with the hostile or odd input that caused it, such as inserted instructions, contradictory documents or missing data. It marks which failures change a reported result.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materiala runnable pipeline and a set of test inputs written by someone other than its builder
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the testers also built the system. Both guidance sources aim at safety and security more than analytic accuracy.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsnone
Kind of findingrisk
Question answeredwhat next
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informingindependent reviewneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakesindependent reviewneededno
high-stakesexpert reviewneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureFollows instructions embedded in source documents.
Error sourcesinput data; language model; expert opinion
Certainty limitsCertainty drops when testers also built the pipeline (risk of bias) and when test inputs cover only known attack types (indirectness).

Evidence and tag basis

Documented. NIST AI 600-1 calls for red-teaming, and the Australian guardrail 4 includes it for general-purpose AI. (NIST AI 600-1; Voluntary AI Safety Standard, 10 guardrails)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.