PJ-11 Agent-based modelling and Monte Carlo simulation (microsimulation)
Simulate agents or random draws many times to give an outcome distribution with tail probabilities and the inputs that account for the spread, as a draft result.
What it does
Simulates individual agents or random draws many times. Artefact: an outcome distribution; read tail probabilities and which inputs account for the spread.
When to use it
Use when the outcome depends on several uncertain inputs or on the behaviour of many actors and you need a distribution, not a single estimate.
When it does not fit
Input distributions are invented or the model was never validated against a historical outcome.
Result format
The result is an outcome distribution from the repeated simulation runs. It gives the tail probabilities and names the inputs that account for most of the spread.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=variable; requires=topic |
| Variable types | Raises a warning | prefers=ratio; prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Behaviour rules or distributions for each input variable and calibration data for them. |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Input distributions are invented or the model was never validated against a historical outcome. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | risk; pattern |
| Question answered | how much; what next |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | held-out validation | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | held-out validation | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | leakage check | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | assumption and limitation statement | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Chooses input distributions without a stated basis and reports tail probabilities as if the model had been validated. |
| Error sources | input data; statistical assumptions; expert opinion |
| Certainty limits | Certainty drops when inputs are invented (risk of bias), when the model was not validated against history (indirectness), or when few runs are made (imprecision). |
Related techniques
- should follow PJ-06 Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate): elicited ranges give defensible input distributions
- pairs with TC1-17 Sensitivity analysis (robustness check, leave-one-out): sensitivity analysis identifies which inputs account for the spread
- should follow CH-01 Key assumptions check (assumptions check): a key assumptions check exposes invented inputs
Evidence and tag basis
Unconfirmed. No source was opened.
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.