Validation
Reading our decoder against public data.
A pipeline that only fits its own runs proves nothing, so here is ours run cold on the field’s published benchmark — their measurement, our decoder.
Key takeaways
- Google published the raw shot data behind its below-threshold surface-code result (Zenodo, CC-BY-4.0). We decoded it with our own pipeline.
- Our decoder reproduces below-threshold behaviour in all eight configurations we decoded — logical error falls with code distance in both bases, with suppression factors Λ between 1.33 and 1.90.
- On identical shots, their best published decoders reach 1.5–2.5× lower logical error than our plain matching decoder — and the gap is the decoder, not the noise model: the two detector-error models agree to within 0.2%.
- Separately, a device model calibrated on one published number blind-predicted a second: Λ = 2.12 against the published 2.14 ± 0.02.
Why decode someone else’s data?
Every quantum team validates against its own runs, and every referee knows how little that proves. The strongest available test is the one you cannot tune: take the field’s published benchmark dataset, run your pipeline on it cold, and print what comes out. Google’s below-threshold experiment ships its raw shots under an open licence — to our knowledge the best public substrate there is for this test.
What did our decoder extract?
Across the eight configurations we decoded, spanning distances 3, 5 and 7 and both bases, our minimum-weight matching decoder reproduces the published result: logical error falls as code distance grows, with suppression factors Λ between 1.33 and 1.90. It also measures our distance to the state of the art: on identical shots, their correlated-matching and neural decoders reach 1.5–2.5× lower logical error than our plain matching. That gap is a work programme, and it is precisely located: our circuit-derived detector-error model and their published one agree to within 0.2%, so the difference is the decoding algorithm, not the noise prior.
Did the model predict a number it had never seen?
A second, independent exercise: the device model we use to set design targets was calibrated against exactly one published number — the distance-7 per-cycle error — and then asked for a number it had never seen, the error-suppression factor. It returned Λ = 2.12; the published value is 2.14 ± 0.02. A model calibrated once and tested blind on a device it was never tuned to is the kind we are willing to design silicon against.
What does this buy?
Our simulations, our decoder and our design model now all touch a refereed third-party result. When Nandi Q1’s design package quotes a predicted logical error rate, the pipeline behind that prediction has been checked against the best public data in the field — with the check itself published, gap included.
Conditions. Dataset: Google Quantum AI’s public surface-code shot data (Zenodo 13273331, CC-BY-4.0) — their measurement on their hardware; we supplied the decode pipeline — detector-error model and matching decoder — and nothing else. Our decode: PyMatching over a Stim circuit-derived detector-error model, 50,000 shots per configuration. Model validation: calibrated on the published d=7 per-cycle error only. Full conditions and job references: the ledger.