You upgraded the model on Tuesday. On Wednesday the agents started producing malformed tool calls. The rollback plan: put the old weights back.
That plan restores one of at least six things that define what your users are actually talking to.
A served model is a release, not a file:
- the weights, and their exact quantization
- the tokenizer and the chat template
- the generation defaults (temperature, stop tokens, max length)
- the inference engine version and its flags
- the system prompts and tool schemas your apps send
- the routing config in front of it (gateway alias, fallbacks)
Change any one and behavior changes. The chat template is the sneaky one: a revision that ships a different template changes how tool calls are formatted with the same weights and a perfect checksum. An engine upgrade can shift outputs with zero model change. And if you serve by a mutable tag ("latest", or any tag a publisher can re-push), a restart can quietly pull a different artifact than yesterday's.
Three things to do before the next upgrade:
1. Write a release manifest. One file per release: weights revision (a commit hash, not a tag), quantization, tokenizer and template hashes, engine image digest, flags, prompt pack version, gateway route. Diffing manifest N against N-1 is your change log and your rollback spec.
2. Do the rollback arithmetic. Illustrative, not measured, arithmetic as of 2026-10-27: a 70B-parameter model in 16-bit is roughly 140 GB of weights (parameters x 2 bytes). Loading that at 1 GB/s from local NVMe takes a couple of minutes. Pulling it over a 1 Gbit link at roughly 0.1 GB/s takes about 25 minutes. Your rollback time is set by storage and network, not by your deploy tool. If N-1 is not staged on the node, "rollback" is a half-hour outage with a polite name.
3. Route through an alias. Apps call "assistant-prod", never a model name. Flipping the alias is the rollback: seconds, as long as N-1 is still resident or staged. That costs GPU headroom. The honest on-prem tax of safe releases is that canary and shadow traffic need VRAM your capacity plan probably did not include.
Gate every swap on your own recorded prompts, comparing rates (tool-call validity, refusals, length, latency), not exact text. Outputs vary run to run, so decide the thresholds before you look at the results.
And an opinion, because I would like you to argue with it: most teams should skip two out of three model upgrades. Every swap costs an eval cycle, a canary and a rollback window. Unless the new model fixes a problem you have measured, an upgrade is risk with a press release attached.
I attached a read-only Python script (standard library only) that hashes a model directory, extracts the chat-template hash, and diffs two release manifests. Review and test it before running it anywhere near production.
Question: how many of the six do you pin today, and which one has bitten you?