Reliable Reports Are a Systems Problem
Trust is not a property of the prose. It is a property of the process that produced it.
The problem
Most attempts to make reports reliable focus on improving the writer: a smarter model, better prompts, a more careful analyst. That approach has a ceiling, because reliability was never a property of writing. A report's worst failure mode is not clumsy prose. It is a wrong claim that nothing catches, and a next edition that quietly forgets the claim was ever made.
So I stopped trying to improve the writer, and built the system around it instead.
The architecture — the writer goes last
A deterministic layer computes the market signals, the contradictions between them, the diff against the previous edition, and event-study backtests of the signals' own history. By the time the language model writes, the analysis already exists. The model renders decisions. It does not make them. It cannot invent a trend the data did not show.
The publication rules — enforced, not aspired to
- Every edition starts with what changed — the engine diff leads, not a fresh summary.
- Every recommendation carries its invalidation conditions — the thresholds that would kill the view, stated at publication.
- Every previous call is scored — against the rules that existed when it was published, never in hindsight.
- Threshold changes are disclosed, not silently rewritten — and scoreboards name the rule set they were scored on.
- Backtests are allowed to weaken the report's own conclusions — inconvenient results are printed.
- Facts, judgment and backtests are tagged apart — and every number traces to a named tool call on a named data series.
These are constraints, not a style guide. The report cannot be produced without satisfying them, so reliability does not depend on anyone's discipline on a deadline day.
The report keeps score on itself
- The first print leaned against us. Headline crude stocks built against the draw thesis. Said plainly, then decomposed: the strategic reserve released more the same week, so the total system still drew.
- Our export paradox resolved against one of our own models. We had named the weekly export print as the tiebreak between two internal models. It arrived. The losing model carries a demerit this edition.
- Our own backtest corrected our own prose. Edition 1 justified a stance with "extremes mean-revert." The backtest found no support in history. The claim was withdrawn in print, and the stance now stands on different grounds.
Where this sits in the landscape
| Layer | The question it answers | Its unit |
|---|---|---|
| Generation (crowded) | How do we generate a better report? | a draft |
| Grounding & provenance (table stakes) | Can I justify this sentence? | a citation |
| Evaluation (growing fast) | Did the model produce a good answer? | an output |
| Publications (this work) | Does this publication deserve trust over time? | an edition |
The first three layers evaluate outputs — a request, a response, a score. A publication is a different unit, because it has the one thing an output does not: memory. It remembers what it claimed, what it scored, and what it corrected. Every mechanism on this page only makes sense for a unit with memory.
How to evaluate a system like this
Not by asking "is this report good?" — a single report is just an assertion. The evaluation is longitudinal: how does the publication behave across editions? Did the calls survive? Were the probabilities appropriately confident? Were corrections disclosed? Reliability is earned across editions, and everything above exists to earn it mechanically.
See it live
The monthly pack
A full risk & strategy report produced under exactly these rules — revision trail, scorecards and appendices included.
View the report →The 5-minute version
Same engine, same rules, one page: the diff, the ledger, the decisions, the view.
View the brief →Methodology & calibration
The full rulebook: every signal predicate, the backtest honesty guards, the revision policy, the scoring rules.
How the reports earn trust →The one-line version: reliable reports do not come from better writers. They come from better publication rules — decided once, enforced by software, and scored in public across editions. The language model writes last. The platform is the editor.