Use by hand
You can follow the whole process yourself, without the skill. This page gives the eight steps in the order the process runs them, with what to do at each one. It is the second way to use startthat.work, next to the skill. It is slower, and several jobs that code does in the skill (ruling out techniques, ordering them, the quote check) fall to you. Nobody has tested the process run this way: the page restates the process and does not add to it.
Each step names the page that explains the step in full.
Step 1: Write the brief
Write your request in one sentence and note its parts. A request such as "do the eight interviews support the three hypotheses about voice AI customer support?" has one part for each hypothesis. Keep your material in one place and note, for each source, its type (text, tables), its size and what one unit of analysis is (a document, a claim, a topic).
Then decide three settings, in the words the process uses.
| Setting | Options |
|---|---|
| Stakes | Exploratory (it decides what to look at next), decision-informing (one input to a decision that can be revisited at moderate cost), or high-stakes (a costly or hard to reverse decision, one that affects rights, safety or finances, or an authoritative publication). |
| Strength of statement needed | Describe. Show a link between things. Predict, without claiming a cause. Estimate the effect of an action. Explain how or why. |
| Kind of finding wanted | Pattern, difference, contradiction, cause, trend, gap, risk or option ranking. More than one can apply, and you can leave it empty. |
The strength of statement matters most. Each technique has a strongest statement it can support, shown on its page. Of the 236 techniques, 177 can describe at most, 23 can show a link between things, 22 can predict without claiming a cause, 9 can estimate the effect of an action and 5 can explain how or why. If you need to say that an action caused an outcome, only the last two groups can support that.
Step 2: Rule out techniques
Open a candidate technique's page and read the table headed Requirements on the material. Five requirements always apply: content type, unit of analysis, level of detail available, number of sources and whether sources can be joined. A technique that fails one of these cannot run on your material. The dictionary explains each property.
Other requirements apply only in some cases, such as the structure of the material, its amount, its language and known data problems. Each applies only when its stated condition is met. The page also states the minimum material a technique needs, for example two or more cases for a comparative case study. The skill's script does not apply minimum amounts yet, so check this line yourself.
If your material has personal information, a technique that cannot take it is ruled out unless a de-identification step will run first.
One exception: a technique that fails only on content type or structure can still run when another technique in your plan produces the input it needs. Its page lists what it needs as input.
Step 3: Choose techniques
Start from the families. Each family page says what its techniques are for. Read the line under each technique name, then its When to use it section. Choose in this order:
- Make sure every part of your request has at least one technique.
- Include at least one technique that reaches the strength of statement you need, if one fits. If none does, say so in your write-up.
- Match the kind of finding you want.
- Where two techniques serve the same part of the request, prefer a pair that reads the material in different ways, because agreement between techniques that read the material in different ways counts for more than agreement between techniques that read it the same way.
Stop at ten, a default chosen for the design and not a measured value. Before you keep a technique, read its When it does not fit section and its Failure modes and limits. Write one line for each technique saying which part of the request it serves.
Each technique carries an evidence status, copied from the sources and never upgraded. Tested means at least one published study measured the technique's effect or accuracy (33 techniques). Documented means a text, official guidance or a summary describes it, without a test (150). Practice means it is used with no published description found (13). Unconfirmed means no source was opened or a source failed to load (40). Treat an unconfirmed technique as a lead for you to check, and say so if you rely on it.
Step 4: Plan the sequence
The kind of technique, shown on its page, sets the order. Question techniques come first, then preparation, then primary techniques. Verification and challenge techniques come last, and they work on the findings, not on your material. Primary techniques do not depend on each other, so they can run in any order. Preparation techniques that feed each other run in the order their inputs require. The kind comes from the technique's job and not from its family. In the stored run, the hypothesis tree is in the family Decompose and map, yet it ran first because it examines the question.
Add to the plan any preparation technique that produces an input a chosen technique needs, with the reason. A technique that needs an additional source (a comparator, a reference structure or an earlier period) says so on its page. Note which ones do. You find the item in step 5.
Then list the library checks. Each technique page has a section, Library checks by stakes, with the minimum checks for your stakes level. Higher levels include the checks of the lower ones. The strength of statement can add more, for example a held-out test for prediction. The check library describes each check.
Step 5: Run the techniques
Run the groups in the order of your plan. Each technique's page has a Result format section that says what its result must contain. Write each result in that format and keep results separate.
- Examine the question. Run the question techniques first. Look at what they find about your request, for example a term that can be read two ways. Then decide: keep the request or edit it. If you edit it, go back to step 2, because different techniques may now fit. Do not choose the question techniques again.
- Prepare the material. Run the preparation techniques. For each technique that needs an additional source, look in your own material first. If the item is not there, find the one item the technique needs, from a source that is not marketing or personal opinion. Record the item, where it came from, its type, the technique that used it and whether you could check its wording against the original. If you find nothing suitable, drop the technique and say why.
- Analyse the material. Run each primary technique on the material. A primary technique works on the material and never on another technique's result.
If a technique's run does not finish, drop it with the reason "did not finish". Do not write a result for it, and list it among the techniques dropped in your answer.
Step 6: Check the findings
Run the verification techniques and the challenge techniques on the findings, then apply the library checks you listed. Challenge techniques examine the findings and never the request.
Checks that need no person you can do yourself. A spot check compares a sample of results with the source. A repeat run runs the technique again and compares the two results. A quote check compares each quoted span with the source text. The skill does this last check by code, and by hand you do it yourself.
Checks that need a person (human review sample, independent review, expert review) apply at high stakes only. At high stakes, have someone review a small sample of the results, and say in your answer whether that happened. Below high stakes, leave them out and say that no person reviewed the results.
Step 7: Combine the findings
Choose a method.
| Method | Use it when |
|---|---|
| Agreement table | Two or more techniques ran and the findings are text or mixed. Rows are the findings and each is marked agree, partly agree, conflict or not covered. |
| Narrative summary | Always, after the agreement table for text or mixed findings. |
| Pooling of estimates | The findings are numeric, and you have confirmed that two or more estimates measure the same thing on the same scale. |
| Decision analysis and weighting | You want an option ranking. |
For each conflict between findings, name the cause or state that the conflict is unresolved.
Then write the conclusions. Each has a title that states the finding as a sentence and an explanation of two to four sentences. Give each one a confidence label and a one-line reason, using this rule.
| Label | When it is given |
|---|---|
| High | Two or more techniques with different methods agree, and their checks passed. |
| Medium | One technique supports the conclusion and its checks passed. Or two or more techniques agree, and a check failed or was not done, or a verification finding limits the conclusion, or the techniques do not have different methods. |
| Low | One technique supports the conclusion and none of its checks passed. Or techniques conflict and the conflict is unresolved. |
Every technique reads the same material, so two techniques are never independent in the material. They count as having different methods when they read it in different ways, for example one counts cases and one searches for statements that contradict a finding. Say what the difference is in the reason. Three further rules apply. A conclusion that depends on an additional source whose wording you could not check is limited to medium. At high stakes, a failed check makes the label low for each conclusion that depends on it. A challenge finding does not change a label: add it next to the conclusion it examines.
A statement about what you did not look at is not a conclusion and gets no label: put it under what was not covered. A statement that the material does not contain something, found by a technique, is a conclusion.
If your request asks for insights, or a conclusion goes against what the request assumed, also write insights. Each states what you or the material would lead a reader to expect, what the analysis found instead, and why the difference matters. A recommendation is not an insight. Give recommendations only when the request asks for them.
Step 8: Write the answer
The answer is the only thing you hand on. It must be complete without your working notes. Write it in this order.
- Summary. The direct answer to the request in two to four short paragraphs of whole sentences, with no technique codes. Say what you found, how far to trust it and what it leaves open.
- Conclusions, in two groups. Put the well supported ones (label high) first, then the ones with weaker support (medium or low). After each conclusion in the second group, add a caveat: one or two plain sentences saying why the support is weaker. Keep the labels and their reasons in your notes and show them only if the reader asks. State once that every technique read the same material and used the same model, which limits how far any conclusion can be trusted.
- Insights, if you wrote any, with a caveat on each one whose label is medium or low.
- What was not covered. The parts of the request the analysis did not answer.
- Strength of statement reached. Which of your techniques reach the strength you needed and which conclusions they support. If none did, say so in the first lines of the answer.
- What the answer is based on. The material you gave, with the type and size of each source. Any additional sources, with the details you recorded. The techniques that ran and what each worked on. The techniques dropped, and why. Any checks not done, and whether a person reviewed the results. Whether you kept or edited the request after step 5.1.
- An offer of a follow-up round. If the reader accepts, run steps 2 to 4 again for the gaps and unresolved conflicts, leaving out techniques that already ran and starting from the techniques you wanted beyond the limit of ten. Then repeat steps 6 to 8 with all the findings. Do not start a follow-up round yourself.
Do not judge whether the material is representative or large enough. You chose it and you know its limits.
An example from the stored run
The simulator replays one run. The request was whether eight interviews support three hypotheses, at decision-informing stakes, wanting to show a link between things, looking for a pattern or a contradiction. Of the 236 techniques, 142 could work on the material. Ten were chosen: the hypothesis tree to examine the question, six primary techniques for the interviews and three techniques to test the findings. The process added one preparation technique, preprocessing sensitivity testing, because stance detection needs a cleaned dataset. The run ended with nine conclusions (four well supported and five with weaker support) and four insights.
What you lose compared with the skill
The skill reads the full list of eligible techniques in one pass, where you work from the families and your own reading. It applies the personal information rule and the requirements for every technique by code, counts the techniques that reach your strength of statement, keeps each technique's result in a fixed format, runs the spot check, the repeat runs and the quote check by code, and records the reason for every choice. By hand, you do these yourself, and a miss in step 2 or step 6 is easy to make. The tags on each technique are inference unless a page names a source, so use them as a guide and not as a test.