Let me start with a photo, like the one in the picture.

Imagine you have a big, sharp photo of a lake and mountains. You print the same photo much smaller, to fit in a wallet. Almost everything is still there: the lake, the trees, the sky. If you hold the two side by side, you see the same picture.

But look closely at the small print. Two tiny dots that were separate in the big photo now sit in the same spot, because the small print has fewer dots to work with. Most of the time nobody notices. Once in a while, those two dots were the part that mattered.

A trained model can be "printed smaller" too. In this lesson I copy one real model into a format called ONNX, which stores its numbers with fewer digits. Then I check, row by row, which answers survived.

This lesson uses the model from the features chapter. If it is new to you, please read what a feature is first. That lesson built six features for each customer of a real online shop. It asked one question: at the start of a month, will this customer buy something in the next 30 days? The model is a gradient boosted tree model from scikit-learn, and it scored a test AP of 0.5450. AP, average precision, is a score from 0 to 1 for how well the model ranks the buyers above the others.

Three lessons of this chapter lead here. Saving formats compared saved the model eleven ways, and every file gave back every prediction to the last bit. It ended by saying a conversion to ONNX would open that question again. Library version skew showed that a saved model often breaks when scikit-learn changes, and mentioned ONNX as a way to run a model without scikit-learn, untested. And container images found that moving from a Mac to Linux changed the very last digit of 33 predictions.

Here I test ONNX for real, on the same model and the same rows.

Please read this slide slowly if any word is new. Every slide after it uses these words.

ONNX (Open Neural Network Exchange) is a file format for models. A pickle file, from earlier lessons, can only be loaded by Python with scikit-learn installed. An ONNX file describes the model with a fixed list of standard steps, so other programs can run it.

A converter turns a scikit-learn model into an ONNX file. The one for scikit-learn is called skl2onnx. A runtime is the program that loads the file and computes answers. I use onnxruntime, which runs on a server with no scikit-learn at all.

A computer stores a number with a fixed number of binary digits, called bits. float64 uses 64 bits and keeps about 16 correct decimal digits. float32 uses 32 bits and keeps about 7. scikit-learn works in float64. ONNX converters usually write float32.

Each tree in the model asks questions like "is money at most 321.65?". The number in the question is a threshold. If the answer is yes, the row goes left; if no, it goes right. The turns one row takes down one tree are its path, and the path decides which leaf value the tree adds.

Before converting anything, I read what the tools themselves say. Every quote here was found word for word in the released source or at a fixed commit, and all of them are in results/onnx-factcheck.json.

skl2onnx's own documentation is open about precision. It says "sklearn-onnx is using single floats by default". Single is another name for float32, and double for float64. It explains that most scikit-learn models compute "with double, not float", and that "the conversion to float introduce small discrepencies compare to double predictions". (The spellings are theirs.)

Then it warns about trees. "In most cases, float and double comparison gives the same result. However, the probability that both comparisons give different results is not null." And: "A decision tree trained for a regression is not a continuous function. Therefore, even a small dx may introduce a huge discrepency." A continuous function changes a little when its input changes a little. A tree does not: a tiny change can send a row down the other branch.

The ONNX specification shows why. The model becomes one step called TreeEnsembleClassifier, whose thresholds are a "list of floats" and whose output is tensor(float), both float32. scikit-learn's own code keeps the input as np.float64. It goes left if data_val <= node.num_threshold, in 64 bits. So the question is: how often does that difference matter, for this model?

Here is the whole trip, in the order it happens.

The converter reads every tree in the model and writes every node into the file: which column it looks at, its threshold, and where to go next. The model has 54 trees with 3,294 nodes, and the file holds all 3,294.

I used the converter's option zipmap set to off. By default, skl2onnx's docs say, it "converts that matrix into a list of dictionaries where each probabily is mapped to its class id or name". A dictionary here is a small lookup table. With zipmap off, the chances come out as a plain table of numbers, one row per customer, which is easy to compare with scikit-learn's.

At prediction time the server sends the rows in, as float32 numbers, and gets the chances back. It never imports scikit-learn.

I wrote the design into the docstring of scripts/labs/packaging/onnx_precision.py before it first ran. Before that I had installed the libraries and read the converter's source code, which told me it writes the thresholds as float32. I had not converted this model.

The model, twenty times. The lab trains lesson 1's model and stops if its test AP is not 0.5450. It also trains it with seeds 1 to 19. A seed is the number that fixes the random choices in training, so each seed gives a slightly different model. The features chapter's rule is to never trust one seed, and here that rule mattered.

Converted two ways. Each model is converted with float32 input, ONNX's usual default, and again with float64 input, to see if 64 bits avoids the rounding.

Compared four ways. For every test row, the lab compares onnxruntime's chance of buying with scikit-learn's. It counts the rows that differ at all and the largest difference. It counts the rows that change place in their month's ranking. It counts the decisions that flip, at a cut-off of 0.5 and at "the top fifth of each month", a rule a shop might use to pick who gets an offer. And it computes test AP, one per month, then the mean.

Located. I wrote my own small walk down every tree, in numpy, that can compare in float64 or float32. It tells me which rows take a different path, and at which split.

The first run never got to a single prediction. Both conversions stopped with an error.

The message was TypeError: Field onnx.AttributeProto.ints: Expected an int, got a boolean. I read the converter's code. For every leaf of every tree, skl2onnx 1.20.0 writes the Python value False into a list that must hold whole numbers. protobuf, the library that writes the file's bytes, refused it.

skl2onnx 1.20.0 was uploaded on 30 January 2026. protobuf 7.34.0 came out on 27 February 2026, a month later, and my venv had protobuf 7.36.2. So I wrote a second design into the docstring, labelled as decided after the first run, and tried three separate venvs on the same saved model.

C2 was the same venv with protobuf 6.33.6, the newest 6.x. It converted. C3 used skl2onnx's unreleased main branch from GitHub, at commit 449ec05, where the line now reads int(nodes_missing_value_tracks_true) # cannot use boolean in onnx. It converted too.

Everything in this lesson runs in C2, the newest combination of released packages that works. C3 comes back later, because it gave different predictions.

This is a real recording of the report script, onnx_report.py, on the laptop where the lab ran.

The report imports nothing from the lab. It checks the sha256, a fingerprint, of every stored array. It retrains seed 0, converts it again, and runs onnxruntime again. It recomputes every comparison for all 20 seeds with its own code and scikit-learn's own AP function. It redoes both bootstraps with the same random draws. And it starts the other venvs again: C1 must still fail the same way, C3 must give the same output, and the small venv on the "What the Server Needs" slide must still give identical output. After an independent review it also reruns two added checks, R1 and R2, explained on the float64 and main-branch slides. All 586 checks agreed.

The report caught one claim of mine. After the main run I wrote down that every changed path belonged to one customer. I had read only part of the output. There were two customers, and the report's check failed until I fixed my sentence. The lab's docstring records the correction.

Here is seed 0, the chapter's own model, converted with float32 input.

Every one of the 26,851 chances of buying was different from scikit-learn's. Not one row was equal to the bit. Compare that with lesson 2, where every saving format gave back all 26,851 to the last bit, and lesson 5, where 33 rows changed in the last digit.

But the differences were tiny. The largest was 1.94e-7, which means 0.000000194, about 2 in 10 million. The average was smaller still, 2.99e-8.

No decision flipped. Not at a cut-off of 0.5, and not at the top fifth of each month. Test AP moved from 0.5449816307508443 to 0.544981591960395, a change of about 4 in 100 million. Per month, AP was equal to the bit in 3 of the 5 test months.

For this model, then, ONNX gave answers that were close everywhere and exact nowhere.

My guess was that most of the difference would come from shorter arithmetic, and that a few rows would take another path. For seed 0, my walk down the trees found no row at all whose path changed. So the whole difference is arithmetic.

The model makes a prediction in two steps. It adds up one leaf value from each of its 54 trees into a raw score. Then a logistic step turns the raw score into a chance between 0 and 1. In ONNX, both steps happen in float32.

I measured the pieces one at a time. Just rounding scikit-learn's final answer to float32 moves it by at most 2.98e-8. Keeping scikit-learn's paths but adding the leaf values in float32 moves it by at most 1.42e-7. And onnxruntime's raw score differed from scikit-learn's by at most 1.06e-6, before the logistic step squeezed that down.

So every row moves, because every row's 54 numbers are added with only about 7 correct digits. A sum of 54 rounded numbers is not the rounded sum.

If every chance moved, did the order of customers change? For seed 0, 22 rows changed their place in their month. I looked at why.

The lab compared every pair of customers in the same month. Not one pair swapped order. If customer A had a higher chance than customer B in scikit-learn, A still had a chance at least as high in ONNX.

What happened instead is ties. 24 pairs that had two different chances in float64 got exactly the same float32 value. When two customers tie, a sort has to put one of them first, and it keeps their order in the table. So a few rows moved by one place, with nobody overtaking anybody.

That matters for any rule that picks "the top N customers". At an exact cut-off, a tie can decide who is in. Here no decision at the top-fifth cut changed.

If I had stopped at seed 0, I would have written "the converter is exact to about 7 digits". Then I looked at all twenty seeds.

For 15 of the 20 seeds, the largest difference stayed under 2.2e-7, just like seed 0. For the other 5 seeds, numbers 5, 11, 12, 14 and 18, it jumped by four to five powers of ten, to between 0.0045 and 0.0507. A difference of 0.05 in a chance of buying is not a rounding detail. It is a different answer.

In each of those five seeds, my walk found 4 or 5 rows that took another path through at least one tree. In the other fifteen it found none. So there are two kinds of difference here, and they are very different in size: the arithmetic, everywhere and tiny, and the path changes, rare and large.

The second series on the chart is the unreleased main branch, which I explain later.

I asked this after the results, and the lab labels it as an addition: which rows changed path, and why?

The largest case was seed 11 and customer 12346, the customer who appears in every lesson of the features chapter, in July 2011. Their money feature was -64.67999999999302. One of the trees asked: "is money at most -64.68?"

In float64, -64.67999999999302 is larger than -64.68, by about 7 trillionths. So scikit-learn said no, and went right. In float32, both numbers round to the same value, -64.68000030517578. So ONNX found them equal, said yes, and went left. The same thing happened in 7 of that model's trees. scikit-learn gave a chance of buying of 0.1444; ONNX gave 0.0937.

Every path change in all 20 seeds was this kind of case, on the money column. In seeds 5, 11, 14 and 18 it was customer 12346, in all five test months. In seed 12 it was customer 17471, whose money was 321.65000000000003 against a threshold of 321.65. Rounding never sends a row the wrong way when the two numbers are far apart. It only collapses two numbers that are almost equal.

Why would a tree have a threshold of exactly -64.68, so close to this customer's value? I asked after the results, and traced it for customer 12346.

In three training months, November 2010 to January 2011, customer 12346's money was exactly -64.68. Those were the only three training rows with that value, and in seeds 5, 11, 14 and 18 the trees used -64.68 as a threshold.

Then, on 18 January 2011, the shop data has two lines for this customer: 74,215 units bought at £1.04, which is £77,183.60, and the same amount returned 16 minutes later. In exact arithmetic they cancel. In float64 they left a trace: from February 2011 on, the sum was -64.67999999999302. The customer's real money did not change; only the last bits of the sum did.

So the threshold was the customer's own old value, and the new value missed it by 7 trillionths, in a direction that only float64 can see. I did not trace customer 17471's history.

The lab also counted this for the seed-0 model. In five of the six columns, every single threshold equals some training value. Frequency is the exception, with thresholds halfway between whole numbers. Test values that land exactly on a threshold are common: 16,965 for return share and 16,675 for products. When both numbers are equal in float64 they are also equal in float32, so with the released converter those rows are safe. The danger is a value a hair above a threshold.

So far: every chance moves a little, and in a quarter of the seeds a few rows move a lot. Did any of it change what the model is used for?

No decision flipped, in any of the 20 seeds, at 0.5 or at the top fifth. The large path changes all happened to customers whose chance of buying was far from either cut-off. Customer 12346, for example, went from 0.1444 to 0.0937, both well below 0.5 and outside the top fifth.

The ranking moved more in the five unusual seeds: up to 4,337 rows changed place in seed 11, because one customer moving down by 0.05 shifts everybody they pass by one place.

Test AP barely moved. The chapter's rule for a question like "is it different?" is a paired bootstrap of the 20-seed mean: draw the test customers again at random, 1,000 times, within each month, and recompute both scores on each draw. The 95% interval of scikit-learn minus ONNX was -1.7e-6 to -1.6e-7. It does not cross zero, so the difference is measurable. ONNX scored higher on average, by less than two millionths.

So, for this model and these rows: measurable, and much too small to change any choice I would make. But one customer's chance moving by 0.05 is a real change for that customer, and in another model, near a cut-off, it would be a flipped decision.

If 32 bits is the problem, why not ask for 64? skl2onnx 1.20.0's classifier converter accepts an input declared as float64. So I tried it on all 20 seeds.

The converter wrote a file for every seed, and the ONNX checker passed it. But onnxruntime refused to load any of them: "Type (tensor(double)) of output arg (probabilities) of node (TreeEnsembleClassifier) does not match expected type (tensor(float))". The file says its output is float64, while this tree step, by the ONNX specification, always outputs float32. Its thresholds were still stored as float32 too. This converter always writes that older step. Its code fixes op_version=1 for this model, and keeps the thresholds in float32 unless the op version is 3 or more.

What would 64-bit input against those 32-bit thresholds have done? onnxruntime would not tell me, so I measured it with my own tree walk, and I label it as a simulation. It was worse than float32. Between 19 and 118 rows per seed took another path, against 0 to 5 with float32. In four seeds a decision at 0.5 flipped, up to 2 in one seed, and one seed flipped 8 at the top fifth.

Here is the reason, and it surprised me. When both the value and the threshold are rounded to float32, a value exactly equal to a threshold stays equal, and goes left, just like in scikit-learn. When only the threshold is rounded, the rounded threshold can land just below the value, and an exactly equal value goes right. For seed 0, all 164 splits where the simulation disagreed had a value exactly equal to the threshold.

After an independent review, I checked whether the format itself can do 64 bits. It can. ONNX's newer step, in version 5 of the operator set, accepts float64 numbers. I built such a file by hand from each model's own trees, with float64 thresholds and leaf values, and onnxruntime 1.30.0 ran it. Over all 20 seeds the largest difference from scikit-learn was 5.6e-16, test AP was equal to the bit in every month, and no decision flipped. Customer 12346 at seed 11 got 0.14439673196282055, against scikit-learn's 0.14439673196282057.

skl2onnx's main branch on GitHub, C3 from the first-run slide, has a change for exactly this model type: pull request #1227, merged on 25 February 2026 and not yet in a release. Its comment says the float64 thresholds "must be adjusted so that float32 comparisons in ONNX give the same routing decisions as sklearn's float64 comparisons."

On my test rows it did the opposite. With main, between 91 and 366 rows per seed took another path, against 0 to 5 with the release. In seed 0, 120 rows changed, the largest by 0.0434, and one decision at 0.5 flipped: customer 15974 in November 2011 went from 0.5021 to 0.4588. Over 20 seeds, main flipped up to 3 decisions at 0.5 and up to 16 at the top fifth in one seed.

I checked what changed in the file. Main lowers 548 of seed 0's 1,620 thresholds: every one whose float32 rounding landed above the true value. The pull request says why: scikit-learn "compares float32 inputs against float64 thresholds". That holds when the data reach scikit-learn as float32. My test rows are float64, and this model compares them in float64, as its _predictor.pyx shows. So a value exactly equal to a threshold goes left in scikit-learn and right past the lowered threshold. In seed 0, all 156 changed splits were exact equalities.

After the review I ran scikit-learn again on the same rows cast to float32 first. Against that, main had 0 rows off by more than 1e-6 in all 20 seeds, and its largest gap was 2.2e-07. The release had 91 to 366 rows off by more than 1e-6, up to 0.11. So main matches scikit-learn when the data reach it as float32, and moves rows when they are float64 and a value sits exactly on a threshold. Neither converter is simply better; which one matches depends on the dtype of your data.

Two honest limits. This is unreleased code at one commit, and it may change before a release. And on test AP, main's 20-seed interval, -3.2e-5 to 3.8e-5, crosses zero: by the chapter's rule it cannot be told apart from customer luck. The harm here is to single rows, not to the average.

Two smaller things turned up after the results, and both are labelled additions in the lab.

Threads. A thread is one worker inside a program; onnxruntime splits the rows across several. I ran the same seed-0 file with 1, 2 and 4 threads. With one thread, 5,731 rows differed from the default run, by at most 3.0e-7. Adding float32 numbers in a different order gives different last digits. The student demo shows the same thing: one row predicted alone gave a slightly different chance than the same row inside the full batch.

Bytes. Converting the same model twice gave two different files. The only difference was the graph's name, a random id such as b3e5ee8ed9b346a5be29a1433449814e. The trees inside were identical. Reproducible artifacts, lesson 6 of this chapter, asks exactly this question for pickle files.

Both point to one habit. Compare predictions with a small tolerance, never with "equal". And compare model files by what they contain, not by their bytes.

The reason to use ONNX at all is the server side. What does loading the file actually need?

I made a second, minimal venv with only onnxruntime 1.30.0 and numpy 2.5.3. The installer added three small helpers that onnxruntime needs: flatbuffers, packaging and protobuf. In that venv, scikit-learn, scipy, pandas, skl2onnx, onnx and joblib all fail to import.

The ONNX file loaded there and gave identical output, to the bit, to onnxruntime in the full venv. The pickle of the same model did not load at all: ModuleNotFoundError: No module named 'sklearn'.

The minimal venv's packages took 105.1 MB on disk, against 447.7 MB for the full lab venv, though the full one also holds pandas and pyarrow, which a server may not need. onnxruntime itself is 78.1 MB of it. The ONNX file was 121,006 bytes, 0.60 times the 202,700-byte pickle. I guessed it would be bigger.

This is ONNX's real promise, and it held: the server no longer depends on scikit-learn's version, which is the whole problem of lesson 3.

I wrote seven guesses into the lab before it ran. Here they are against the results.

1. "skl2onnx 1.20.0 converts this scikit-learn 1.9.1 model without error." Wrong. It failed with protobuf 7, and worked with protobuf 6.

2. "f32: almost every row differs from scikit-learn by a tiny amount, max abs difference below 1e-6 on rows whose path did not change; some rows change path because x32 == t32, and those differ by more, up to about 0.05. I guess fewer than 1% of rows change path." Right. Every row differed, by at most 2.2e-7 where the path did not change; at most 5 rows changed path, by up to 0.0507.

3. "f32: test AP equal to 4 decimal places, not to the bit." Right.

4. "f32: at most 5 decisions flip at 0.5, at most 20 at the top-fifth cut; more than 100 rows change rank within their month (mostly new ties among float32 values)." Right on flips: none. Wrong on rank for seed 0: 22 rows. Right that ties, not swaps, caused it.

5. "f64: the converter accepts float64 input, but the thresholds are still float32 in the file, so some rows still change path (fewer than f32), and the output is still float32." Half right: it converted with float32 thresholds, but onnxruntime would not load it, and my simulation found more path changes, not fewer.

6. "the ONNX file is bigger than the pickle, between 1.5x and 3x." Wrong: 0.60 times.

7. "the minimal venv predicts bit-identical to the full venv's onnxruntime, and cannot load the pickle." Right.

The full lab trains twenty models, uses four venvs and runs a bootstrap. I wrote a small demo that shows the core of it with one Python.

I wrote the demo's design into its docstring after the lab had run and before the demo first ran. After its first run I changed three small things, and the docstring says so. The one that matters: it had predicted customer 12346's row on its own and got 0.09372258186340332, while the lab, predicting all rows at once, got 0.09372246265411377. The demo now predicts all rows and prints both.

Before you run this lab. You need Python 3 with pandas, pyarrow, scikit-learn, skl2onnx, onnx and onnxruntime. These are the versions I used, with the one pin that matters: pip install scikit-learn==1.9.1 skl2onnx==1.20.0 onnx==1.23.1 onnxruntime==1.30.0 "protobuf<7" pandas pyarrow openpyxl. I ran it inside the lab's venv, which you can make the same way and activate with source ~/lab-data/venv-pkg-onnx-pb6/bin/activate. First run python fetch_data.py from the scripts/labs/features folder; it downloads the shop data once, about 46 MB.

Then, inside the examples folder, run python onnx_demo.py. I ran it on a Mac with an Apple processor. These libraries run on Windows and Linux too, but I have not checked the numbers there; the last digits may differ, as lesson 5 found for Linux.

This box holds the real results from the lab for all 20 seeds. It needs nothing but Python, so it runs in your browser. It loads no model.

Press Run. It rounds customer 12346's money and the threshold to float32 the way ONNX stores them, and shows which way each side goes. Then it prints seed 11's results. Change X, T or SEED and run again. Try X = -64.6799 to see a gap that float32 can still see.

The report script writes this box from the lab's stored results and checks what it prints. The function f32 uses Python's struct module to store a number in 4 bytes and read it back, which is exactly the rounding ONNX does.

The lab is one file, scripts/labs/packaging/onnx_precision.py. It runs in the C2 venv.

data imports lesson 1's own feature code and builds the train and test rows. train fits the model for one seed. convert calls skl2onnx with float32 or float64 input and zipmap off, and run_ort runs a file with onnxruntime.

compare takes scikit-learn's chances and another set of chances. It returns everything on the headline slides: rows that differ, the largest and mean difference, rank changes, flips at both cut-offs, and AP per month. ap_one computes AP in exactly scikit-learn's order of operations, and the lab checks it gives the same bits as scikit-learn's function.

tree_tables and walk are my own tree walk. They read each tree's nodes and send every row down it, comparing in float64 or float32. adj32 copies the main branch's threshold rule, so the walk can predict which rows main moves. first_disagreements finds, for each changed path, the first split where the two comparisons disagree, and records whether the value equalled the threshold.

Here is how I would convert a model like this one, using only what this lab measured.

The chart turns on one question, asked after the comparison: did any decision flip, or is one row's gap far bigger than all the others? Here are the same steps in words, each with the number from this lab behind it.

1. Pin the whole set. skl2onnx, onnx, onnxruntime and protobuf, together. Here a protobuf release a month after skl2onnx's broke the conversion. Lesson 4 of this chapter shows how to lock versions.

2. Use a released converter, and match the dtype. Convert with float32 input, its default; with skl2onnx 1.20.0's classifier converter, float64 input gave a file that would not load. If your rows are float64, the release matched scikit-learn better here; if they are float32, main's change targets your case. One option I did not test: cast your data to float32 at the source, before both training and serving, so every tool sees the same numbers.

3. Keep test rows and scikit-learn's chances next to the file. After converting, run onnxruntime on those rows and compare three things: the largest difference, the order of customers, and every decision your system makes from the chance.

4. Look at the worst row. If one row differs by far more than the rest, find its path. A value a hair above a threshold is the usual cause here.

5. Test more than one model. Seed 0 had no path change at all; 5 of 20 seeds did. If you retrain often, compare every new model, not just the first.

Use ONNX when the server should not depend on scikit-learn. Here the file ran in a venv a quarter the size of the full one, and it did not care which scikit-learn trained it. That removes the whole problem of lesson 3.

Use it when tiny differences are fine for every decision you make. Here no decision flipped in 20 seeds with the released converter.

Do not use it blindly for a model whose decisions sit near a cut-off. One customer's chance moved by 0.05 here. Had it been near 0.5, it would have flipped.

Do not expect skl2onnx's float64 option to fix precision. With its 1.20.0 classifier converter it gave a file that would not load. The format can do 64 bits, through the opset-5 TreeEnsemble that onnxruntime runs, but this converter does not write it.

The right converter depends on what dtype your data are in. On these float64 rows the release moved 0 to 5 rows per seed and the unreleased main branch 91 to 366. On the same rows cast to float32, main matched scikit-learn in every seed and the release did not.

Do not compare ONNX output with scikit-learn's using equality. Every row differed, by design of the format.

One model type. Every number here is for the chapter's gradient boosted model. A model with more trees adds more rounded numbers, and skl2onnx's docs say the gap "grows with the number of trees in the forest". A linear model or a neural network would behave differently.

One Mac, one CPU runtime. onnxruntime has other ways to run a model, such as on a graphics card. I did not test them.

No timings. The load average was 3.92 when the lab started and 8.90 when it ended, because other work was running. Speed is a common reason to use onnxruntime, and I measured none.

Released code and one commit. C2 is released code. C3 is GitHub main on one day; a released version may behave differently.

Float64 was tested through skl2onnx 1.20.0's converter. Its file would not load, so those float64 numbers come from my own tree walk. After the review I also built a float64 opset-5 TreeEnsemble by hand, which onnxruntime ran to within 5.6e-16. I did not test any converter that writes that step.

Labelled additions. Five questions were asked after the main results: the changed rows, the threads, the bytes, how often values sit on thresholds, and customer 12346's history. The three-venv test was designed after the first run failed. All are labelled in the lab, the report and here, along with the one claim the report corrected.

If you take one thing to work on Monday, find a model you already serve as ONNX, or plan to. Run it next to the original on a few thousand stored rows and print the largest difference. If it is around 1e-7, the arithmetic is the only difference. If it is much bigger, find that row and the split where its path changes.

Then write the comparison into your build, with a tolerance, so it runs every time the model, the converter or the runtime changes.

The one idea to keep: a model converted to ONNX is a new model that is close to the old one everywhere and equal to it nowhere. Here the closeness was enough for every decision, but a value seven trillionths above a threshold moved one customer by 0.05. Test the converted model on stored rows, on more than one seed, before it replaces the original.