TogetherAI Launches ThunderAgent to Cut Agentic Inference Thrashing, Citing 2.5x Higher Throughput
TogetherAI introduced ThunderAgent, an inference scheduler designed to resolve cache thrashing during agentic AI workloads, claiming 2.5x higher single-node throughput and roughly 10x lower P50 latency at high concurrency.
The system addresses a workload cycle where interrupted agent tool calls trigger cache evictions and subsequent full recomputations. By pausing lower-priority tasks when memory is constrained and routing resumed operations to available nodes, the research was selected for a Spotlight paper at ICML 2026.
From the sources (4 posts)
@togethercomputeAgentic inference wastes GPUs on KV cache thrashing. ThunderAgent fixes it at the scheduler level: 2.5x higher single-node throughput and ~10x lower P50 latency at high concurrency. ThunderAgent was accepted to ICML 2026 as a Spotligh
@togethercomputeThe result is a vicious cycle. Agent A pauses for a tool call, its cache gets evicted, the tool returns, the engine recomputes Agent A's whole history from scratch, which evicts Agent C. Cascade. This is KV cache thrashing.
@simran_s_aroraRT @GT_HaoKang: Super excited to see our work ThunderAgent helpout with TogetherAI agent RL/serving products. Meanwhile, ThunderAgent has…
@togethercomputeWith that abstraction it does program-levels scheduling: under memory pressure it pauses low-priority workflows so the rest hit far higher cache-hit rates. Resumed workflows route through a global queue to the node with the most free ca