Microsoft-Decision-1, explained: a model that scores answers instead of writing them, and a test to run

Decision-1 runs only as a hosted API. Microsoft's card says the weights are not distributed, so there is nothing to download and run locally. Calling it needs a Microsoft Foundry project or an OpenRouter account with credit, and this machine has neither, so I did not call it. No result on this page comes from Decision-1. Every measured number names the model that produced it.

What anyone can check without an account is OpenRouter's public listing. It shows one provider, Azure, a 32,768-token limit, an input price of $0.000000042 per token ($0.042 per million) and an output price of zero. The modality reads text to decisions: there is no text output at all.

The claim that matters most for real use is the robustness one. If a model's answer changes when you only change the order of the options, its probabilities are partly about the order, not the ticket. Microsoft says Decision-1 never flipped when options were reversed or shuffled. That is a test you can run yourself, so I built it.

I wrote 24 short support tickets and labelled each with the queue I would send it to: billing, technical, account or sales. A few are deliberately borderline, like an invoice PDF with a layout bug. Each ticket went to two models twice, once with the options in order and once reversed.

The first model is qwen2.5:3b, a plain 3-billion-parameter chat model, running in Ollama (a free tool that runs models on a laptop) on my Mac. I listed the options as letters A to D, asked for one letter, and read the probability the model gave each letter in that single step (its logprobs). That is the cheap way to turn any chat model into a scorer. It is an older, much smaller model from the same Qwen family Decision-1 is built on; Decision-1 was trained further for this job, so this result says nothing about Decision-1 either way. The second is Jev, TypeSafe's hosted decision model, which works the way Decision-1 does: a choice question in, a probability per option out.

The plain model changed its answer on 4 of the 24 tickets when the options were reversed. All four were sales tickets, and three of them moved to billing. Ticket t19, a team of 400 asking for volume pricing, came back sales at 100% one way and billing at 100% the other.

The raw responses show why. On three of the four the model answered D, the last letter, in both orders. Going forward D was sales; reversed, D was billing. The letter stayed the same and its meaning changed underneath it. That is position bias: part of the answer comes from where an option sits, not from the ticket.

That is the dangerous part. A cut-off (a probability below which you send the case to a person) would not have caught any of the four, because the model was certain both times. In fact 47 of its 48 answers came back at 99.5% or more, so for this model a cut-off was useless everywhere. Its probabilities said how sure it was, not how right it was. Jev changed none of its 24 answers, and on average its probabilities moved by 0.5 percentage points between the two orders.

What this test does not show: 24 tickets is a small test, I wrote and labelled them myself, I only reversed the options (Microsoft also shuffled and reworded them), and neither model is Decision-1. The two runs also differ in more than the model: the small one saw letters, Jev saw named options, and Jev is trained for this job and probably much bigger. So the qwen2.5:3b flips show the test catches a real failure; the gap between the two models does not show which of those differences caused it.

Jev matched my label on all 48 calls, but my tickets are easy for a model trained for this job. The result does not say Jev or Decision-1 is accurate on your traffic. It says the test is cheap and it catches a real failure, so run it on your own questions before you set any threshold.

Read next: Cascades (a cheap model first, hard cases to a bigger one): choosing which questions to pass on