Steps 6 to 8: Check, combine and answer
Step 6: Check the findings
Three things happen, in this order.
- Verification techniques run on the findings of the analysis, for example a search for disconfirming statements or a sensitivity analysis.
- Challenge techniques run on the findings of the analysis, for example devil's advocacy or a premortem. They never challenge the brief.
- Library checks are applied to each result, as set by the stakes and the strength of statement.
A library check that needs no person is done by the process, by code or by running the technique again. That includes spot checks and repeat runs. Human review sample, independent review and expert review always need a person. Gold standard agreement needs a person unless the material already has labels made by trained people.
Checks that need a person apply at high stakes only. Below high stakes the plan leaves them out, because a wrong conclusion is cheap to find and correct and the check would not change any label. The answer then says that no person reviewed the results. At high stakes the process stops at the end of step 6, shows the user a small sample of the results with the labels or findings the model produced, and the user reviews it, skips it or defers it. Reviewing the sample tests the output, not the decision points inside the process, so it takes minutes and not the time of redoing the analysis.
Step 7: Combine the findings
The process offers the methods that apply and recommends one. It runs the recommended method if the user does not reply.
| Method | Applies when | Recommended when |
|---|---|---|
| Agreement table (agree, partly agree, conflict, not covered) | Two or more techniques ran. | Findings are text or mixed. |
| Narrative summary | Always. | Findings are text or mixed, after the agreement table. |
| Pooling of estimates | The user confirms that two or more estimates measure the same thing on the same scale. | Findings are numeric and the user has confirmed. |
| Decision analysis and weighting | An option ranking is wanted. | An option ranking is wanted. |
For each conflict between findings, the process names the cause or states that the conflict is unresolved.
Each conclusion gets one confidence label (high, medium or low) and a one-line reason. The label is recorded for every conclusion. The answer groups the conclusions by it and shows the label itself only when the user asks (decision 46). The rule is a design choice, not a published method, and is untested.
| Label | When it is given |
|---|---|
| High | Two or more techniques with different methods agree, and their checks passed. |
| Medium | One technique supports the conclusion and its checks passed. Or two or more techniques agree, and a check failed or was not done, or a verification finding limits the conclusion, or the techniques do not have different methods. |
| Low | One technique supports the conclusion and none of its checks passed. Or techniques conflict and the conflict is unresolved. |
Different methods. Every technique in a run reads the same material and is run by the same model, so two techniques cannot be independent in data or model. They count as having different methods when they read the material in different ways, for example one counts cases and one searches for disconfirming statements. The model states the difference in the reason. Two techniques tagged as substitutes for each other do not count. The answer states the shared material and model as a limit.
Checks that need a person. They apply at high stakes only. A check that the user did not review is named in the reason and limits the label to medium until a person does it. Below high stakes the check is left out and does not change the label.
Additional sources. A conclusion that depends on an additional source whose wording code could not check, for example a summary returned by a fetch tool, is limited to medium, and the reason names the source.
Failed checks at high stakes. At high stakes a failed check makes the label low for each conclusion that depends on it, and the reason names the check.
Challenge findings do not change the label. A challenge finding is an added finding and appears in the answer next to the conclusion it examines.
Titles and scope. Each conclusion has a title that states the finding as a sentence, an explanation of two to four sentences, the label and a reason in plain words. A statement about what the process did not look at is not a conclusion and gets no label. It goes in the answer under what was not covered. A statement that the material does not contain something, found by a technique, is a conclusion and gets a label.
Insights. When the request asks for insights, or when a conclusion goes against what the request assumed, step 7 also writes insights. Each states what the request or the material would lead a reader to expect, what the analysis found instead, and why the difference matters. An insight gets a label by the same rule as the conclusions it rests on. A recommendation is not an insight. The answer gives recommendations only when the request asks for them.
Step 8: Answer
The answer is the only output of the process. Everything produced in steps 1 to 7 is work.
- Where the application can create a document, the answer is delivered as a document, so that the user can keep, share and export it. Otherwise it is given in the reply (decision 44).
- The answer is always given to the user. It must be complete without the work, so it names the techniques that ran and the checks that were not done.
- The work is kept. How much of it the user sees depends on the application that runs the process: it can show the work as it happens, show it on request or not show it.
- Where the process asks the user to decide something, it always shows what the user needs for that decision: the brief at intake, the findings from 5.1, the methods offered in step 7 and the gaps behind a follow-up round.
- The user can change the chosen techniques only in an application that shows them.
The answer has these parts, in this order.
- Summary. The direct answer to the request, in two to four short paragraphs of whole sentences with no technique codes. It says what the analysis found, in the order a reader needs it, how far to trust it and what it leaves open. A reader who has seen nothing else must be able to understand what was found.
- Conclusions, in two groups. Well supported conclusions (label high) come first, then conclusions with weaker support (label medium or low). Each shows its title and its explanation. In the second group each also carries a caveat: one or two plain sentences saying why the support is weaker. The conclusions have no codes, and the answer shows the labels and their reasons only when the user asks for them. Each group is introduced by the fixed words below.
- Insights, when step 7 wrote any, introduced by the fixed words below. An insight whose label is medium or low carries a caveat in the same way.
- What was not covered. The parts of the request the analysis did not answer, and what the process did not look at. These have no label.
- Strength of statement reached.
- What the answer is based on.
- The offer of a follow-up round.
Fixed words for the groups of conclusions. Well supported: "Two or more techniques that read the material in different ways agree on each of these conclusions, and the checks on them passed." Weaker support: "The analysis supports these conclusions less well. After each one, a sentence says why." Limit, added to the first group shown: "All the techniques read the same material and used the same language model, which limits how far any conclusion can be trusted."
Fixed words for the labels, used only when the user asks to see them: "Confidence shows how well the analysis supports a conclusion as it is written. It does not show how important the conclusion is. High means two or more techniques that read the material in different ways agree, and their checks passed. Medium means one technique supports it, or several agree but a check failed or was not done, or they read the material in the same way. Low means one technique supports it and its checks did not pass, or the techniques conflict and the conflict is not resolved. All the techniques read the same material and used the same language model, which limits every label."
Fixed words for the insights: "An insight is a finding that goes against, or beyond, what your question or the material would lead a reader to expect. Each one states the expectation and what the analysis found instead."
Parts 5 and 6 are stated as facts.
Strength of statement reached. Which chosen techniques reach the strength of statement the user needed, and which conclusions they support. When none did, the answer says so in its first lines.
What the answer is based on.
- What the user gave: each source, its type and size.
- Any additional sources used: the item, where it came from, its type, which technique used it, and whether code could check its wording.
- Which techniques ran and what each worked on.
- Which techniques were dropped, and why.
- Any files that could not be read.
- Any checks that were not done, and whether a person reviewed the results.
- Whether the user confirmed the request after 5.1.
The answer can also include a section on the process followed: the techniques chosen, the sequence they ran in and the recorded reasons for both. Whether the answer includes this section by default is not yet decided (section 10).
The answer makes no judgement about whether the material is representative or sufficient, because the user chose the material and knows its limits.
Follow-up round. The process offers one. If the user accepts, steps 2 to 4 run again for the gaps and unresolved conflicts found in step 7, leaving out techniques that already ran and starting from the reserve list. The new techniques run, and steps 6 to 8 are repeated with all the findings.
The methods in step 7 come from reviews of how to combine evidence from studies. Using them to combine the findings of techniques within one run is an extrapolation, and only abstracts and summary pages were read.