Talkie Debuts as Open-Weight LLM Built on Pre-1931 Text
A team including Alec Radford and David Duvenaud has introduced Talkie, an open-weight historical language model with 13 billion parameters trained on 260 billion tokens of English text from before 1931. The developers said the model is intended to help researchers study how language models generalize, including whether systems grounded in older writing can be adapted to tasks such as coding.
Examples shared alongside the release highlighted both the model’s historical voice and its limits. Asked about the causes of a new world war, Talkie answered that Germany would seek to regain lost provinces and re-establish itself as a major continental military power. Other tests suggested the system, while knowledgeable through 1931, can remain anchored to early-20th-century scientific ideas, defending the aether hypothesis and distrusting special relativity; one reported exchange also showed it wrongly placing Hitler’s rise as dictator and control of the Reichswehr in late 1929, though those events occurred in 1933 and 1934.
From the sources (12 posts)
@alecradRT @DavidDuvenaud: Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset o…
@alecradRT @status_effects: New work with @AlecRad and @DavidDuvenaud: Have you ever dreamed of talking to someone from the past? Introducing talk…
@thom_wolfRT @DavidDuvenaud: Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset o…
@danshipperif a new world war were to break out, what would be its causes? "Germany would strive to regain lost provinces and establish herself as a great military power on the Continent"
@pmarca😍
@emollickThis is an incredibly cool experiment It is also fascinating that the model knows information up to 1931, but, at least in some science topics, seems very stuck in the early 1900s. For example, it defends the lumiferous aether hypothesis
@pmarcaProjects like this open up an entire world that has been wiped from our collective memory. Like discovering a new planet, populated by totally different people.
@boazbaraktcsRT @status_effects: New work with @AlecRad and @DavidDuvenaud: Have you ever dreamed of talking to someone from the past? Introducing talk…
@deredleritt3rInterestingly, Talkie knows that Hitler was a "dictator" and that he "got possession of the Reichswehr" - even though these events happened in 1933/1934. When questioned, Talkie claims that this all happened in late 1929.
@soumithchintalaRT @DavidDuvenaud: Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset o…
@simonwSome notes on talkie, a new "vintage language model" from a team including Alec Radford (yes, that Alec Radford) "trained on 260B tokens of historical pre-1931 English text"
@boazbaraktcsRT @DavidDuvenaud: I’m interested in historical LLMs for backtesting long-term forecasting methods. Right now it’s hard to evaluate long-t…