SOOFI AI Model Recreates Nvidia Open-Source Stack and Leaks Benchmark Questions
A research thread argues that the SOOFI consortium’s new SOOFI-S model is a direct clone of Nvidia’s open Nemotron-3-Nano stack, trained on existing public data and using 2T tokens more than the original. The analysis claims the model relies entirely on Nvidia’s architecture and training code, rather than developing a sovereign foundation.
Critics allege benchmark scores are inflated because the training dataset, created by partner TU Darmstadt, contains rephrased evaluation questions the model is tested on. The consortium claims sovereign frontier status using a custom metric not used across the broader industry, though the dataset is now reportedly being cleaned to remove test leakage. The launch announcement omitted any reference to the Nvidia stack, contradicting its stated transparency.
From the sources (8 posts)
@jjitsevWhat happened: SOOFI consortium, led by same orgas responsible for previous openGPT-X and Teuken, presents a model which was trained using already existing, open Nemotron 3 data and stack. SOOFI-S a Nemontro-3-Nano clone, trained from scrat
@jjitsevFirst big overclaim is the stated sovereignty. SOOFI-S is a copy of Nemotron 3, using to large extent already existing open data and training code made by NVIDIA. Sovereignty requires understanding about design (data, arch, training, scalin
@jjitsevThe "sovereignty" statement in current form strongly misleads the public. The announcements do not even mention building on Nemotron-3-Nano stack, making public believe the effort is self-made. This stands in strong contrast to claims of "r
@jjitsevSecond big overclaim is the "frontier-level" "champion" status. SOOFI invents its own benchmark, "capability index", used by noone else, and argues with the scores to match or even outperform the original Nemotron-3-Nano they clone (using 2
@jjitsevIssue 2: Soofi trains on a dataset (AIML-TUDA/QA-base) that contains rephrased sets of most of evals used for comparison. . Nemotron-3-Nano training does not use rephrased evals in this form and quantity. The compar
@jjitsevThe dataset was made for SOOFI by TU Darmstadt, a partner. Following the hints, it is now being cleaned from test leakage ( original version used for SOOFI training Nemotron 3 Nano uses much
@jjitsevSOOFI thus compares a Nemotron-3-Nano clone that has seen training sets of most of its evals (and in one case, GPQA, the test set), to original Nemotron-3-Nano that has not. Claiming "frontier-level champion" status does not work this way,
@jjitsevIn summary, instead of measuring true performance, SOOFI thus presents eval leakage contaminated numbers to claim "frontier-level champion" status, claiming "sovereignty" for execution of an already existing open Nemotron stack, where actua