Large Corpus Pain Map

In short

A large corpus cannot be honestly summarized with a single prompt: the model sees only part of the messages and easily mistakes a local slice for the overall picture.

Outcome

A recursive pass covers the entire recorded slice, preserving sources and disagreements, and the resulting pain map separates confirmed clusters from hypotheses.

How the automation runs

Trigger

The corpus exceeds the model's working window, and an aggregating conclusion for the entire slice is required

Automation steps

1. Fixes corpus version, volume, and rules for removing service noise
2. Divides material into stable chunks and processes each by one rubric
3. Collects interim conclusions with pointers to original messages
4. Recursively combines clusters, preserving rare signals and contradictions
5. Checks coverage and releases only conclusions with traceable examples

Human check

The researcher approves the rubric, checks a sample of quotes, and decides which clusters to consider a product opportunity; sensitive corpus data is not sent to an unauthorized service.

Outcome

A recursive pass covers the entire recorded slice, preserving sources and disagreements, and the resulting pain map separates confirmed clusters from hypotheses.

Using it

When to use it

You need to get a distribution of themes or pains across a corpus that is actually larger than the context window, rather than finding a single fact.

How to verify

For each cluster, original messages are found, the covered slice is indicated in the report, and a selective check confirms quotes and preserved disagreements.

Tools

Recipe details