Summaries that improve
without manual tuning
A closed-loop system that evaluates session summary quality with a strong LLM, identifies weaknesses, generates improved prompts, validates them against real data, and promotes winners automatically.
Static prompts don't scale
Summarizer prompts are written once and applied to every session — regardless of complexity, language, domain, or team conventions. Some summaries are excellent. Others miss critical decisions, overemphasize boilerplate, or produce descriptions too vague for code review.
Evaluate. Optimize. Validate. Promote.
Five stages that close the gap between what summaries produce and what they should produce.
Summarize
The session summarizer runs as usual, but tags every output with prompt provenance — which prompt version was used and a fingerprint of the input context.
Evaluate
A strong model (Claude or GPT-4o) scores sampled outputs across multiple dimensions. Each of the 11 fact categories is scored independently with detailed reasoning.
Optimize
The evaluator analyzes patterns in low- and high-scoring outputs, then generates a revised prompt that targets identified weaknesses.
Validate
The candidate prompt is tested against real sessions. It must demonstrably outperform the current prompt before promotion. Marginal gains are flagged for human review.
Monitor
After promotion, outputs are continuously scored. If quality drops below historical baselines, the system automatically rolls back to the previous prompt.
Scored at two levels
Every sampled output is evaluated at the chunk level and the summary level — measuring extraction completeness, accuracy, and hallucination independently per fact category.
Chunk-level evaluation
For each chunk: did the extraction capture all visible facts? Are they accurate? Did the model hallucinate anything not in the context? All 11 categories scored independently.
Summary-level evaluation
Does the final summary accurately reflect the session? Are key decisions complete and well-stated? Are the tags meaningful for filtering and search?
Built for safe, measurable improvement
Prompt provenance
Every output is tagged with the exact prompt version and input context hash. Full traceability from output to instructions.
Database-driven prompts
Active prompts load from a registry. Update prompts without redeploying. Full version history with parent lineage.
Auto-rollback
If a promoted prompt causes quality regression, the system detects it and rolls back automatically. No manual intervention needed.
Quality trends
Track extraction completeness, accuracy, and hallucination rates per category over time. See which dimensions improve and which need attention.
Lineage limits
Configurable depth limit on automatic iterations. The system pauses for human review after N generations, preventing unbounded prompt drift.
TTL-managed storage
Evaluation contexts expire automatically via ClickHouse TTL. Prompt history and scores are kept permanently — they're small and valuable.
Automatic or human-in-the-loop
Start with evaluation-only to build confidence in the scores, then switch to auto when ready. Both modes can coexist.
Automatic
The system manages the full loop: evaluate, generate improved prompts, validate, promote winners, rollback failures. Hands-off continuous improvement.
Best for teams that want improvement without dedicating engineering time to promptsManual
The evaluator provides quality scores and detailed feedback. Engineers write improved prompts informed by the data. The system handles validation and promotion.
Best for teams that prefer direct control over prompt changesNo coupling. Full traceability. Safety first.
No coupling
Summarizer and evaluator communicate only through ClickHouse. Either can be deployed, upgraded, or removed independently.
Graceful degradation
The summarizer works with code defaults if no prompt registry exists or the evaluator is never deployed.
Full traceability
Every output links to its prompt version and input context. Every prompt links to its parent. All optimization decisions are logged.
Cost-bounded
Strong model evaluations run on configurable sample sizes (default 20) on a periodic schedule. One candidate prompt per run. Predictable costs.
Summaries that get better on their own.
See the documentation for configuration and deployment options.