Ship only what beats the baseline.
BrainstormLLM tries to predict which phases of a software-development pipeline a task actually needs, from specification through documentation, so AI agents can skip the rest. It was trained on 2,203 real Claude Code session turns and had to clear kill gates set before any training began.
Better. Not good enough.
A one-shot model reached a mean F1 of 0.52 under five-fold cross-validation. Conditioning each phase on the phases before it lifted that to 0.61, and above 0.8 on whether a task needs a specification or an implementation at all. Review, verify and deploy stayed below 0.45: whether a task needs them depends on what happened during implementation, which the request alone cannot show.
Even 0.61 is generous. It gives each phase the true outcome of the phases before it, which a live pipeline only knows after the fact.
Kill gate: mean F1 of at least 0.75
Set in advance. Held to.
The thresholds were written down before the first model was trained, and missing any one of them keeps the model out of the request path. The accuracy gate failed, so it stayed there.
| Gate | Threshold | Result | Status |
|---|---|---|---|
| Enough data | 200+ examples, 3+ task types | 2,203 / 9 | Pass |
| Phase-skip accuracy | Mean F1 of at least 0.75, five-fold cross-validated | 0.61 | Fail |
| Cost and quality | At least 25% cheaper than running every phase, with no more than a 5% quality drop | Not measured | Open |
Sequential predictors
Gradient-boosted models, one per pipeline phase, each conditioned on the phases before it.
Model-tier classifier
Trained separately on 400,000+ RouterBench data points; 93% cross-validated accuracy.
Inline inference
Both models export to ONNX, small enough for BrainstormRouter to call inline once a model clears its gates.
Part of one loop.
The control plane records full pipeline runs, build and test results included. That is the data the open quality gate needs, and the next training set.