Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

History LLMs: Models trained exclusively on pre-1913 texts

github.com

Post reply on HN
Print view View original

41–50 of 452 posts

Re: History LLMs: Models trained exclusively on pre-1913 texts

#41

Post by tootyskooty »

mmooss wrote: post #30
zozbot234 wrote: post #27

Earlier quoted context omitted.

They apparently pre-train with all data up to 1900 and then fine-tune with 1900-1913 data. Anyway, the amount of available content tends to increase quickly over time, as instances of content like mass literature, periodicals, newspapers etc. only really became a thing throughout the 19th and early 20th century.

They pre-train with all data up to 1900 and then fine-tune with 1900-1913 data. Where does it say that? I tried to find more detail. Thanks.

See pretraining section of the prerelease_notes.md:

https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

Re: History LLMs: Models trained exclusively on pre-1913 texts

#42

Post by jimmy76615 »

> We're developing a responsible access framework that makes models available to researchers for scholarly purposes while preventing misuse.

The idea of training such a model is really a great one, but not releasing it because someone might be offended by the output is just stupid beyond believe.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#44

Post by tonymet »

I would like to see what their process for safety alignment and guardrails is with that model. They give some spicy examples on github, but the responses are tepid and a lot more diplomatic than I would expect.

Moreover, the prose sounds too modern. It seems the base model was trained on a contemporary corpus. Like 30% something modern, 70% Victorian content.

Even with half a dozen samples it doesn't seem distinct enough to represent the era they claim.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#45

Post by tonymet »

Heliodex wrote: post #4

The sample responses given are fascinating. It seems more difficult than normal to even tell that they were generated by an LLM, since most of us (terminally online) people have been training our brains' AI-generated text detection on output from models trained with a recent cutoff date. Some of the sample responses seem so unlike anything an LLM would say, obviously due to its apparent beliefs on certain concepts, t…

the samples push the boundaries of a commercial AI, but still seem tame / milquetoast compared to common opinions of that era. And the prose doesn't compare. Something is off.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#46

Post by derrida »

I wonder if you could query some of the ideas of Frege, Peano, Russell and see if it could through questioning get to some of the ideas of Goedel, Church and Turing - and get it to "vibe code" or more like "vibe math" some program in lambda calculus or something.

Playing with the science and technical ideas of the time would be amazing, like where you know some later physicist found some exception to a theory or something, and questioning the models assumptions - seeing how a model of that time may defend itself, etc.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#47

Post by gbear605 »

briandw wrote: post #12

So many disclaimers about bias. I wonder how far back you have to go before the bias isn’t an issue. Not because it unbiased, but because we don’t recognize or care about the biases present.

I don't think there is such a time. As long as writing has existed it has privileged the viewpoints of those who could write, which was a very small percentage of the population for most of history. But if we want to know what life was like 1500 years ago, we probably want to know about what everyone's lives were like, not just the literate. That availability bias is always going to be an issue for any time period where not everyone was literate - which is still true today, albeit many fewer people.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#48

Post by _--__--__ »

joeycastillo wrote: post #22

A question for those who think LLM’s are the path to artificial intelligence: if a large language model trained on pre-1913 data is a window into the past, how is a large language model trained on pre-2025 data not effectively the same thing?

You're a human intelligence with knowledge of the past - assuming you were alive at the time, could you tell me (without consulting external resources) what exactly happened between arriving at an airport and boarding a plane in the year 2000? What about 2002?

Neither human memory nor LLM learning creates perfect snapshots of past information without the contamination of what came later.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#49

Post by mediaman »

mmooss wrote: post #28

> Imagine you could interview thousands of educated individuals from 1913—readers of newspapers, novels, and political treatises—about their views on peace, progress, gender roles, or empire. I don't mind the experimentation. I'm curious about where someone has found an application of it. What is the value of such a broad, generic viewpoint? What does it represent? What is it evidence of? The answer to both seems to…

This is a regurgitation of the old critique of history: what's it's purpose? What do you use it for? What is its application?

One answer is that the study of history helps us understand that what we believe as "obviously correct" views today are as contingent on our current social norms and power structures (and their history) as the "obviously correct" views and beliefs of some point in the past.

It's hard for most people to view two different mutually exclusive moral views as both "obviously correct," because we are made of a milieu that only accepts one of them as correct.

We look back at some point in history, and say, well, they believed these things because they were uninformed. They hadn't yet made certain discoveries, or had not yet evolved morally in some way; they had not yet witnessed the power of the atomic bomb, the horrors of chemical warfare, women's suffrage, organized labor, or widespread antibiotics and the fall of extreme infant mortality.

An LLM trained on that history - without interference from the subsequent actual path of history - gives us an interactive compression of the views from a specific point in history without the subsequent coloring by the actual events of history.

In that sense - if you believe there is any redeeming value to history at all; perhaps you do not - this is an excellent project! It's not perfect (it is only built from writings, not what people actually said) but we have no other available mass compression of the social norms of a specific time, untainted by the views of subsequent interpreters.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#50

Post by kazinator »

> Why not just prompt GPT-5 to "roleplay" 1913?

Because it will perform token completion driven by weights coming from training data newer than 1913 with no way to turn that off.

It can't be asked to pretend that it wasn't trained on documents that didn't exist in 1913.

The LLM cannot reprogram its own weights to remove the influence of selected materials; that kind of introspection is not there.

Not to mention that many documents are either undated, or carry secondary dates, like the dates of their own creation rather than the creation of the ideas they contain.

Human minds don't have a time stamp on everything they know, either. If I ask someone, "talk to me using nothing but the vocabulary you knew on your fifteenth birthday", they couldn't do it. Either they would comply by using some ridiculously conservative vocabulary of words that a five-year-old would know, or else they will accidentally use words they didn't in fact know at fifteen. For some words you know where you got them from by association with learning events. Others, you don't remember; they are not attached to a time.

Or: solve this problem using nothing but the knowledge and skills you had on January 1st, 2001.

> GPT-5 knows how the story ends

No, it doesn't. It has no concept of story. GPT-5 is built on texts which contain the story ending, and GPT-5 cannot refrain from predicting tokens across those texts due to their imprint in its weights. That's all there is to it.

The LLM doesn't know an ass from a hole in the ground. If there are texts which discuss and distinguish asses from holes in the ground, it can write similar texts, which look like the work of someone learned in the area of asses and holes in the ground. Writing similar texts is not knowing and understanding.

Post reply on HN