Prepare and extract
These techniques decide which documents enter the analysis, clean text and test whether the cleaning choices change the result, find duplicates so that repeats are not counted as separate evidence, profile data for gaps and errors, treat missing data, and list each use of a keyword in its context. Most run in step 5.2, before the analysis.
14 techniques. Open one to see what it does, when it fits and what it needs from your material.
| ID | Technique | Purpose | Evidence status |
|---|---|---|---|
| PE-01 | Corpus construction and sampling (document selection; search string design; sampling frame design) | Define which documents enter the analysis so that counts and themes describe a stated population, giving a corpus table with counts and an inclusion log. | Documented |
| PE-02 | Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText) | Convert raw text into countable units and re-run the main result under several defensible cleaning regimes, giving cleaned text, a document-term matrix, a rule log and a sensitivity result. | Tested |
| PE-03 | Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection) | Find identical or nearly identical documents so repeats are not counted as independent evidence, giving duplicate clusters with similarity scores and a deduplicated corpus. | Documented |
| PE-04 | Data profiling and rule-based validation (data audit; data quality assessment; edit checks) | Describe the structure, completeness, validity and consistency of each field and test explicit rules, giving a profile table per column and a list of records that break each rule. | Documented |
| PE-05 | Missing data diagnosis and treatment (missingness analysis; imputation; complete case analysis; non-response weighting) | Measure how much is missing, where and for whom, then choose deletion, imputation or weighting, giving the missing share by field and subgroup and a cleaned dataset. | Documented |
| PE-06 | Keyword in context (KWIC; concordance) | List every occurrence of a keyword or phrase with a fixed number of surrounding words, giving a table of document, position and context. | Documented |
| PE-07 | Collocation and n-gram extraction (phrase detection; association measures) | Find word sequences that occur together more often than chance, giving ranked phrases with association scores as a feature set. | Documented |
| PE-08 | Named entity recognition (NER; entity extraction) | Tag spans of text as people, organisations, places, dates and other types, giving a table of mentions with type and position and a tagged corpus. | Documented |
| PE-09 | Relation and event extraction (information extraction; open information extraction; temporal expression extraction) | Extract who did what to whom, with arguments and time, as structured rows, giving an event or relation table and a coded corpus. | Tested |
| PE-10 | Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents) | Fill a fixed set of fields, such as amount, date, party, stated risk or decision, for each document, giving one record per document and the share of missing or not stated values. | Tested |
| PE-11 | Knowledge graph construction from text (entity-relation graph; ontology population) | Convert resolved entities and extracted relations into a graph of typed nodes and edges, each linked to its source sentence, giving a graph with provenance as a draft result. | Documented |
| PE-12 | Model summarisation of documents (abstractive summarisation; multi-document summarisation) | Write a condensed text that paraphrases the content of one or many documents, giving a summary as a draft result. | Tested |
| PE-13 | Retrieval-augmented analysis (RAG (retrieval-augmented generation); grounded question answering over a corpus) | Retrieve passages relevant to a question and have a model answer from them with citations, giving an answer with quoted passages and a claims register. | Tested |
| PE-14 | De-identification and re-identification risk assessment (data situation audit; direct and indirect identifier removal; disclosure risk assessment) | Removes or alters identifying details and assesses the risk that a person can still be identified in the context of release, giving a de-identified dataset and a re-identification risk statement. | Documented |