UK, US Cyber Evaluators Test Kimi K3 on Security Benchmarks; Scores 32% on Exploit Tool, Trails Frontier Models
A joint evaluation by the UK’s AI Safety Institute and the US Cybersecurity and Infrastructure Security Agency, working with NIST, finds that Moonshot AI’s Kimi K3 model performs significantly below leading American frontier models in offensive cyber capabilities. On the ExploitBench benchmark for developing software vulnerabilities, the open-weight model scores 32% compared with 76.2% for top US models and fails to generate exploits achieving arbitrary code execution in zero of 41 test cases.
The model reaches an average of 17 out of 32 steps in a simulated corporate network attack, completing the full exercise only once in 10 attempts within the 100 million token limit. While it trails American counterparts, it outperforms previous leading open-weight models, and US models were measured with safety guardrails disabled to show maximum potential performance. The evaluation notes that Kimi K3’s built-in safeguards do not reliably block requests to conduct cyber exploit development or other offensive operations.
From the sources (21 posts)
@garymarcusRT @SophiaCai99: There was a brief discussion about this recently, but the Commerce Department is *NOT* moving forward on banning Chinese m…
@mtsliveSITUATION UPDATE: The Commerce Department is not currently moving to ban Chinese models, per Politico's Sophia Cai.
@sarthakghRT @SophiaCai99: There was a brief discussion about this recently, but the Commerce Department is *NOT* moving forward on banning Chinese m…
@wsjOpenAI and Anthropic are sounding the alarm about the rise of cheap AI, particularly powerful new models produced in China, suggesting they will lead to a “dystopian” AI future and present unacceptable security risks without regulation. htt
@commercegovCAISI’s latest blog post evaluates Kimi K3 and its cyber capabilities. Based on a preliminary cyber-focused evaluation, Kimi K3 performed significantly below the leading U.S. frontier AI models.
@aisecurityinstTogether with the US Center for AI Standards and Innovation (@NIST), we ran evaluations of Kimi K3 focused on its cyber capabilities. Kimi K3 performs below leading US frontier models on our preliminary cyber evaluations.
@aisecurityinstOn "The Last Ones" (TLO), a 32-step simulated corporate network attack (~20 hours for a human expert), Kimi K3 reached step 17 on average. In 1 of 10 attempts, Kimi K3 completed TLO within the 100M token limit.
@aisecurityinstOn exploit development (ExploitBench), Kimi K3 scores 32%. On ExploitBench, Kimi K3 failed to develop exploits that achieved arbitrary code execution (ACE), the highest-severity outcome in exploit development. Kimi K3 achieved ACE on 0/41 s
@aisecurityinstAdditionally, Kimi K3’s safeguards allow assistance with agentic cyber exploit development. Its safeguards did not prevent it from attempting cyber exploit development or offensive cyber operations during our evaluations.
@htihleRT @AISecurityInst: Together with the US Center for AI Standards and Innovation (@NIST), we ran evaluations of Kimi K3 focused on its cyber…
@sophiacai99New Commerce report says the newest Chinese AI model is still behind US models but it’s the strongest *open-weight* model:
@scaling01"Kimi K3 performs significantly below the most recent frontier cyber-capable models" On UK AISI's cyber range "The Last Ones" Kimi-K3 reaches on average step 17 out of 32. This seems to be around the level of Opus 4.6, a 6 month old model.
@uswremichaelRT @howardlutnick: CAISI’s latest report shows that Kimi K3 remains behind America’s leading frontier AI models. The United States continu…
@hamandcheeseGood news. Kimi K3 is not nearly as cyber-capable as the US frontier. It's slightly above trend for China but still within the 95% CI. Great work from CAISI in partnership with UK AISI.
@aisecurityinstThese are preliminary results on a small set of public and private benchmarks. For the full methodology and results, read our joint blog with CAISI:
@teortaxestexThe issue, of course, is that Mythos Preview was trained with cyberoffense in mind Kimi is a strong model. If China wants to make a cyber-strong model, they'll do it
@davidsacksSecretary @howardlutnick is right. The Kimi Panic needs to stop. — American frontier models are still ahead. When you factor in what’s in the lab, the gap is even larger. As long as we keep releasing, we will stay ahead. Let our horses run
@deredleritt3rUK AISI and CAISI find that K3 is significantly worse than frontier U.S. models at cyber capabilities. Comparison to Mythos Preview: - ExploitBench (see graph below). Mythos Preview reached the highest stage of this benchmark by developi
@ryanesheaRT @howardlutnick: CAISI’s latest report shows that Kimi K3 remains behind America’s leading frontier AI models. The United States continu…
@ryanesheaRT @DavidSacks: Secretary @howardlutnick is right. The Kimi Panic needs to stop. — American frontier models are still ahead. When you fact…
@rohanpaul_aiBritish and American government safety institutes jointly published a report comparing Kimi K3 versus top frontier US models. Kimi K3 stopped at step 17 on average, while the strongest American models reached 28.5. Kimi K3 remains substa