Reproducibility is the feature financial firms are actually buying
Non-deterministic output is the single most common reason an AI pilot fails risk review. The fix is engineering discipline, not a better model.
Ask a risk officer why an AI pilot stalled and you will rarely hear "accuracy." You will hear that the same input produced a different answer twice, and nobody could explain the difference. In a regulated firm, that ends the conversation.
Full determinism is not available from a sampling language model. Bounded, explainable variation is — and that is what a control framework requires.
What to pin
Reproducibility comes from removing every source of drift you control:
- Pin model versions explicitly. Never point production at a floating alias.
- Version prompts in the repository and reference them by hash in the trajectory.
- Set temperature to zero for extraction and classification. Reserve sampling for drafting.
- Constrain outputs to a schema and reject anything that fails validation rather than repairing it silently.
- Version the retrieval index and record which version served each run.
- Make tool contracts explicit and stable, with typed arguments and typed responses.
- Keep deterministic computation out of the model. Ratios, dates, and thresholds belong in code.
Move the arithmetic out of the model
The highest-leverage change in most financial workflows is narrowing the model's job. Let the model find and extract the number and cite where it came from. Let ordinary code compute the ratio, compare it to the covenant, and decide whether it breached.
This is not a limitation of current models. It is good system design: a deterministic computation should be performed deterministically, and the result should be identical every time by construction.
Prove it with a regression suite
Build a golden dataset of real cases with verified expected outputs, including the awkward ones. Run it on every prompt change, every model change, and every index rebuild, with explicit pass thresholds and a hard gate on release.
The suite is also your evidence pack. When someone asks how you know the workflow performs, you have a versioned result set rather than an opinion.