Kalai et al (2025) Further Statistical Study of NISQ Experiments

Re-examine Google’s 2019 supremacy claim and three newer NISQ demos. They cannot reproduce the famed “Formula (77)” fidelity predictions; discrepancies reach 40–50 % on patch sub-circuits and read-out data are internally inconsistent. Conclude that none of the published NISQ experiments (Google, Quantinuum, Harvard/QuEra, USTC) supply enough raw data or statistical transparency to be considered conclusive.

Paper: “Further Statistical Study of NISQ Experiments”
Authors: Gil Kalai, Tomer Shoham, Carsten Voelkmann (Hebrew University & TU Munich)
Venue: arXiv 2512.10722 (Dec 2025)

Transcript

1. Q UA N T U M S U P R E M A C Y D E B A T E DYB Musings 2019-2025 Google's Quantum Supremacy Claims
2. NAVIGATION Contents 01 The Quantum Supremacy Landscape 02 Google's 2019 Sycamore Experiment 03 The Formula (77) Controversy 04 Kalai et al.'s Critical Analysis 05 Competing Claims & Replications 06 The NISQ Era Context 07 Implications & Future Directions
3. The Quantum Supremacy Landscape
4. CHAPTER 01 What is Quantum Supremacy? John Preskill's Definition (2012) Quantum supremacy describes the point where a programmable quantum device can perform computations that would be extremely difficult to carry out on conventional supercomputers , regardless of practical utility. Supremacy vs. Advantage Quantum Supremacy Quantum Advantage The NISQ Era (2018-Present) NIntermediate-Scale Quantum devices with 50-1000 qubits , characterized by high error rates and limited coherence times. The arena for all quantum supremacy claims to date. Timeline of Key Events 2012 Preskill Coins "Quantum Supremacy" Oct 2019 Google's Sycamore Claim 2019-2020 IBM & Academic Skepticism 2021-2024 USTC Zuchongzhi Series 2023-2025 Kalai et al. Statistical Analysis Dec 2024 Google's Willow Chip
5. CHAPTER 01 The NISQ Era: Promise and Limitations NISQ Characteristics 50-1000 ~10⁻³ ~1000 μs-ms No Error Correction: Insufficient resources for full quantum error correction. Gate Fidelities: Single-qubit ~99-99.5%, Two-qubit ~95-99%. Major Players & Roadmaps Google Quantum AI Error-corrected quantum computer by 2029 IBM Quantum Quantum-centric supercomputer by 2025 USTC (China) Zuchongzhi 3.0 6 orders of magnitude Quantinuum Target: Fault-tolerant computing by 2030 The Fundamental Challenge The Gap: Current quantum computers have not demonstrated practical advantages for real-world problems beyond contrived benchmarks. Error rates remain too high for most proposed algorithms.
6. Google's 2019 Sycamore Experiment
7. CHAPTER 02 The Original 2019 Claim The Sycamore Processor 53 20 ~200s Architecture: Superconducting quantum processor with simultaneous operation of all qubits. Technology: Sycamore chip featuring 54 qubits (one malfunctioned, leaving 53 active). The Computational Task Random Circuit Sampling (RCS) Generating random bitstrings of length 53 from a quantum probability distribution. The quantum computer applies random quantum gates, then measures the output state. Verification: Cross-Entropy Benchmarking (XEB) fidelity measures how closely the experimental samples match the ideal quantum distribution. The Dramatic Comparison Google's claim was compared to Wright Brothers' first flight, Sputnik, Moon landing, Fermi's nuclear chain reaction, Higgs boson discovery, and LIGO gravitational waves —positioning it as a civilization- altering achievement. The Supremacy Claim ~200s ~10,000 years Key Technical Details Qubit Count: 53 active qubits Circuit Depth: 20 cycles of random gates Samples: 1 million bitstrings generated XEB Fidelity: ~0.2% (small but statistically significant) Classical Verification: Would require computing 2⁵³ amplitudes
8. CHAPTER 02 IBM's Immediate Counter-Arguments The 10,000 Year Dispute IBM argued Google's 10,000-year estimate was off by 10 orders of magnitude . Optimized classical algorithms could perform the same task in approximately 2.5 days. ~10,000 years ~2.5 days The Circuit Change Allegation IBM claimed Google changed the circuit type just weeks before the final experiment after developing more sophisticated classical algorithms. This suggested Google avoided circuits where classical simulation was more efficient. IBM's Philosophical Position IBM views "quantum supremacy" as unimportant compared to "quantum advantage" for practical problems. Their focus is on demonstrating quantum computers can solve useful problems better than classical computers. The Usefulness Critique Critics noted that random circuit sampling has no practical application . It's an artificial problem designed to be hard for classical computers, not to solve real- world challenges. John Preskill's Nuanced Assessment Preskill noted that Google's achievement is significant in demonstrating the team understands its hardware and that it is working. However, he acknowledged IBM's point: what was demonstrated can also be done on classical computers, albeit more slowly. Stripped of hype, this was an important step in controlling quantum systems, not a revolutionary breakthrough in computation.
9. The Formula (77) Controversy
10. CHAPTER 03 Understanding Formula (77) The Formula F̂ = ∏(1-eg) × ∏(1-eg) × ∏(1-eq) G₁: Set of 1-qubit gates | eg: Error probability G₂: Set of 2-qubit gates | eq: Readout error Purpose & Significance A priori prediction of circuit fidelity based on individual component error rates . If accurate, it demonstrates understanding and control of the quantum system. Google's Claimed Accuracy The predicted fidelity values were within 10-15% of experimental XEB measurements —a remarkably close match given the complexity. Average Error Rates Reported 1-Gate Errors 0.16% 2-Gate Errors 0.62% Readout Errors 3.8% Simplified Approximation F̂* = (1-0.0016)|G₁| × (1-0.0062)|G₂| × (1-0.038)n
11. CHAPTER 03 The Statistical Surprise Why 10-15% Deviation Was Surprising With hundreds of independent error sources, random error accumulation should produce larger deviations. The match suggested either remarkable precision in error characterization or systematic optimization . Google's Three Justifications 1. Accurate Individual Error Estimation 2. Unbiased Estimation Errors 3. Statistical Independence The Deviation Estimation Formula DEV = 0.2 × √(n×0.038 + |G₁|×0.0016 + |G₂|×0.0062) For n=53, |G₁|=795, |G₂|=301: DEV ≈ 8.6%. The Surprise Google reported only 10-15% deviation between Formula (77) predictions and experimental XEB fidelities. 10-15% The Skeptical View Kalai et al. argued that assumptions 2 and 3 are implausible . The calibration process represents a global optimization toward expected outcomes, invalidating statistical claims. The Core Issue If individual error estimations have 20% uncertainty, the cumulative deviation should be larger. The 10-15% match suggests systematic bias or optimization.
12. Kalai et al.'s Critical Analysis
13. CHAPTER 04 The Implausibility Argument Kalai, Rinott & Shoham (2023) Core Skepticism: The remarkable predictive power of Formula (77) is statistically implausible under standard assumptions about quantum noise. optimization process that invalidates the statistical claims The Calibration Process Key Finding: The calibration process represents a global optimization toward expected outcomes , not an unbiased characterization. Implication: The Implausibility of Assumptions Assumption 2: Unbiased Errors Skeptical view: systematic biases are expected Assumption 3: Statistical Independence Skeptical view: Gate and readout errors are likely correlated The Burden of Proof Kalai et al. argue that the burden of proof lies with demonstrating these assumptions hold , not assuming them without verification.
14. CHAPTER 04 Data Retrieval and Analysis Attempting to Verify Formula (77) Kalai et al. sought to independently verify Google's Formula (77) predictions using Google's own data. Google had not provided the individual 2-gate fidelities. Data Sources Obtained Individual Readout & 1-Gate Errors 2-Gate Errors: The Missing Data did not provide individual 2-gate fidelities RGB Color Extraction Method extracted 2-gate errors from Figure 2b The Key Finding "We could not reproduce the values obtained from Formula (77) reported in the Google paper." Deviations were substantial. The RGB Extraction Process 1 Read RGB values for each tick mark (0.0008 to 0.017). 2 Interpolated 99 RGB values between each tick mark. 3 Assigned error values based on minimal L1-distance. Data Request Timeline Initial requests Partial data provided Additional readout data 2-gate data still missing Implications The inability to reproduce Formula (77) predictions raises fundamental questions about Google's methodology and the validity of the supremacy claim.
15. CHAPTER 04 Inconsistencies in Google's Data Inconsistent Readout Error Data Google provided two different readout error datasets (2020 and 2021) with substantial gaps between them , casting doubt on the claim that error rates are stable within 20%. Qubit-Dependent Average Errors Kalai et al. discovered that average readout error rates changed as a function of the number of qubits, contradicting Google's claim of stability. The Patch Circuit Anomalies Google's experiment included "patch circuits" on two non-interacting sets of qubits. XEB fidelities were systematically ~10% lower than Formula (77) predictions . The puzzle: If Formula (77) is correct, patch circuits should yield predictions matching individual patches, but they don't. Readout Error Inconsistencies 2020 Data: som_params_by_qubit.csv 2021 Data: readout_raw_data.tar Patch Circuit Results XEB: 0.583 XEB: 0.577 F̂: 0.583 Implications These inconsistencies suggest the experimental system is not as well-characterized as claimed, and the predictive success of Formula (77) may be due to factors other than genuine a priori prediction.
16. CHAPTER 04 Kalai's 2025 Findings 1 Failure to Reproduce "We have not been able to reproduce (or even come close to reproducing) the values obtained from Formula (77)" using Google's own data. 2 Reduced Prediction Quality Using individual readout errors reduced prediction quality compared to rougher average-based estimates. 3 Mismatched Predictions Combined data did not match Google's reported Formula (77) values or empirical XEB fidelities. The Deviation Analysis Reported vs. Computed the computed Formula (77) predictions differed substantially The Individual Error Effect increased the deviation Kalai et al.'s Conclusion "The assertions of [Google's 2019 paper], even those referring to the 10-20 qubit range, cannot be considered conclusive."
17. Competing Claims & Replications
18. CHAPTER 05 USTC's Zuchongzhi Series The Zuchongzhi Progression University of Science and Technology of China (USTC) has been systematically advancing quantum supremacy claims , competing directly with Google. 2.1 3.0 3.0 Zuchongzhi 3.0 Specifications 105 182 Coherence Time: 72 μs Single-Qubit Fidelity: 99.90% Two-Qubit Fidelity: 99.62% Readout Fidelity: 99.13% Commercial Deployment (2025) China deployed Zuchongzhi 3.0 for commercial use via the "Tianyan" quantum cloud platform , attracting over 37 million visits from 60+ countries. The 2025 Achievement 83-Qubit, 32-Cycle RCS ~Few Hundred Seconds 6.4×10⁹ Years Comparison to Google USTC's experiment achieved a quantum advantage 6 orders of magnitude higher than Google's latest. 10⁶× Global Implications
19. CHAPTER 05 Google's Willow Chip (2024-2025) The Willow Announcement December 2024: Google announced Willow, a 105-qubit superconducting chip , achieving a major milestone in quantum error correction. The Benchmark Achievement ~5 min 10²⁵ years Below-Threshold Achievement The 30-Year Challenge: Willow demonstrated exponential error reduction as qubit counts increased —going "below threshold." Error Correction Scaling 3×3 Array 9 5×5 Array 25 7×7 Array 49 Significance for Fault Tolerance "Below-threshold" behavior is essential for building large-scale fault-tolerant quantum computers. Implication: Large, error-corrected quantum computers can be constructed.
20. CHAPTER 05 Other NISQ Experiments Quantinuum's Trapped-Ion Experiments Platform: H2 trapped-ion quantum processor Method: Random geometry circuits with flexible connectivity Sample Size: 20-100 bitstrings (vs. Google's 500K) Kalai's Concern: Small samples don't allow testing empirical distribution Harvard/QuEra Neutral Atoms Achievement: Logical circuits with neutral atoms Recognition: Among the most important breakthroughs Data Request: Kalai et al. requested data in 2024 Finding: Strong stability of Fourier-Walsh transform General Methodological Concerns Small Samples: Don't allow proper statistical verification Many Circuits: Enable optimization toward outcomes Limited Data: Insufficient transparency for verification Core Issue: Harder to detect optimization with many different circuits The Certified Randomness Debate The Promise Kalai's Critique Cannot be trusted
21. The Broader Debate
22. CHAPTER 06 The Quantum Computing Skeptics The Broader Skeptical View Beyond Google's claims, a minority of physicists and computer scientists question the fundamental possibility of scalable quantum computing. Gil Kalai's Core Argument "Empirical probability distributions...are combinations of noise-sensitive distributions that are useless for computation and computationally very primitive robust distributions." Implication: cannot support scalable quantum information Robert Alicki's Thermodynamic Argument "No-Go Theorem": Thermodynamics implies that the set of effectively stabilized states for a large system always possesses a classical structure . Key Skeptics Gil Kalai Robert Alicki Michel Dyakonov Dyakonov's Critique Building quantum computers is a "ridiculously hard technological task" because the threshold theorem relies on impossible exact assumptions. The Skeptics' Challenge Skeptics argue that gradual experimental progress will hit a barrier, and fantastical claims are unlikely to withstand rigorous examination.
23. CHAPTER 06 The Proponents' View The Mainstream Perspective The majority of physicists and computer scientists view quantum computing as an engineering challenge, not a fundamental impossibility . The BQP Argument BQP (Bounded-error Quantum Polynomial time) characterizes the complexity of quantum processes in nature. Implication: classical simulation of quantum physics is fundamentally hard The Experimental Evidence Quantum Supremacy: Multiple demonstrations by Google, USTC Steady Progress: Increasing qubit counts and fidelities Error Correction: Successful prototypes (Google Willow, IBM) Key Proponents John Preskill Scott Aaronson Industry Leaders Response to Skeptics Proponents argue that quantum mechanics has been verified to exquisite precision for a century. The burden of proof is on skeptics The Engineering Challenge Building quantum computers is extremely difficult, but no fundamental physical principle prevents it . With sustained investment, error-corrected quantum computers will be built.
24. CHAPTER 06 Certified Randomness and Trust The Certified Randomness Promise Goal: Generate random bits that are certifiably not from a pseudorandom generator , even if the source is adversarial. Application: The Method 1. Run quantum supremacy experiment 2. Verify classical computation is intractable 3. Use quantum outputs as certified random bits Quantinuum & JPMorganChase 2025 Proposal: Use Quantinuum's trapped-ion processor for certified randomness generation, as published in Nature. Kalai's Critique Problem 1: Extrapolation Risk Extrapolation is heuristic Problem 2: Classical Simulation classical simulation rather than genuine quantum computation The Trust Problem For an adversarial agent, it's relatively easy to present fake quantum supremacy claims. Fundamental Issue: certify the validity of quantum supremacy claims themselves Implication Certified randomness based on quantum supremacy cannot be trusted until we can verify the quantum claims themselves.
25. Implications & Future Directions
26. CHAPTER 07 Methodological Lessons Transparency Complete data disclosure is essential. Individual fidelities, raw measurements, and calibration procedures must be provided. Statistical Rigor Avoid optimization toward outcomes during calibration. Independence and unbiasedness assumptions must be validated. Reproducibility Detailed experiments and large samples are needed even in the 10-20 qubit range to enable independent verification. Small-Scale Validation If claims cannot be verified for small systems, extrapolation to large systems is suspect . Broader Implications for Experimental Science Complex Systems hundreds of interacting components Commercial Pressures intense commercial competition Verification Burden transparent, reproducible evidence
27. CHAPTER 07 The Path Forward Recommendations for Researchers Detailed Small-Scale Experiments 10-20 qubit range Detecting Optimization Bias detecting systematic optimization Classical Simulation Benchmarks properly benchmark quantum claims For the Community Rigorous Standards: Establish standards for supremacy demonstrations. Independent Verification: Create protocols for independent verification. Separate Hype from Reality: Maintain scientific objectivity. Unresolved Questions Can QEC Scale? Practical Quantum Advantage? How to Verify Claims? The Three-Step Roadmap 1 Quantum Supremacy 2 Quantum Simulators 3 Fault-Tolerant Computers
28. CHAPTER 07 Concluding Assessment The Engineering Achievement Google's 2019 claim represented a significant engineering achievement in controlling 53 qubits simultaneously—a major advance in experimental physics. The Statistical Concerns However, the statistical methodology and data transparency raise legitimate concerns that cannot be dismissed. The Nature of Scientific Debate This debate reflects healthy scientific skepticism in a rapidly evolving, commercially charged field. The Unresolved Question The Proponent View stepping stone to fault-tolerant systems The Skeptical View approaching fundamental limits The Path Forward The burden of proof remains high for extraordinary computational claims. Future progress requires greater transparency, independent verification, and rigorous separation of engineering achievement from computational advantage claims . The quantum computing field must address skeptics' concerns rather than dismiss them.
29. The Quantum Question "The question of whether quantum computation is possible is among the most important open scientific questions of our time." — Gil Kalai, 2025 Transparency Rigor Skepticism