Routing AI Requests Without Training Data and Knowing When to Train a Classifier

The review sampling policy becomes part of the training data once uncertain cases and disagreements are deliberately oversampled. That is useful for learning boundaries, but the resulting label distribution no longer represents ordinary traffic.

I would retain the sampling reason and, where possible, selection probability with each reviewed request. Keep a separate randomly sampled traffic slice for estimating auto-dispatch error and coverage at deployment prevalence. Otherwise a classifier can improve on the carefully curated hard-case set while the reported operating error moves simply because the review policy changed. The same distinction helps compare the cascade with a trained replacement fairly.

The point about labels being a snapshot is the one I'd stress for the later classifier stage. When you do start collecting labels, keep the capability catalog version next to each one and hold out a test set per version, so a drop in accuracy after onboarding a new agent shows up as a routing regression and not as noise. Class imbalance will also bite early: a forecasting agent that gets 2% of traffic can look fine on overall accuracy while failing most of its requests, so report per-route recall alongside the abstain rate. A small hand-written set of the ambiguous multi-domain questions, like your sales-are-down example, makes a useful fixed regression suite.