Google Launches JAXBench TPU Benchmark Showing Curated Documentation Boosts AI Agent Correctness to 37.3%

Google, Harvard University and the University of California, Berkeley have introduced JAXBench, a benchmark of 50 JAX workloads built from established machine learning architectures to evaluate how artificial intelligence agents optimize TPU kernel code. Testing with the Gemini 3 Flash model showed that conditioning on curated technical documentation raised per-sample correctness from 5.8% to 37.3%.

The benchmark evaluates generated code against pre-tuned kernels from Tokamax rather than basic baselines, highlighting documentation and search methods as the main constraints for models navigating unfamiliar programming interfaces. The setup also yielded a 1.28x geometric mean speedup across benchmarks, extending to 1.36x when combined with beam search, underscoring that structured context matters more than model size for AI coding agents.