Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

History LLMs: Models trained exclusively on pre-1913 texts

github.com

Post reply on HN
Print view View original

221–230 of 452 posts

Re: History LLMs: Models trained exclusively on pre-1913 texts

#221

Post by hackinthebochs »

sodafountan wrote: post #166
frahs wrote: post #97

Wait so what does the model think that it is? If it doesn't know computers exist yet, I mean, and you ask it how it works, what does it say?

It would be nice if we could get an LLM to simply say, "We (I) don't know." I'll be the first to admit I don't know nearly enough about LLMs to make an educated comment, but perhaps someone here knows more than I do. Is that what a Hallucination is? When the AI model just sort of strings along an answer to the best of its ability. I'm mostly referring to ChatGPT and Gemini here, as I've seen that type of behavior wit…

LLMs are extrapolation machines. They have some amount of hardcoded knowledge, and they weave a narrative around this knowledgebase while extrapolating claims that are likely given the memorized training data. This extrapolation can be in the form of logical entailment, high probability guesses or just wild guessing. The training regime doesn't distinguish between different kinds of prediction so it never learns to heavily weigh logical entailment and suppress wild guessing. It turns out that much of the text we produce is highly amenable to extrapolation so LLMs learn to be highly effective at bullshitting.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#222

Post by physicsguy »

p-e-w wrote: post #207
bondarchuk wrote: post #204

> Historical texts contain racism, antisemitism, misogyny, imperialist views. The models will reproduce these views because they're in the training data. This isn't a flaw, but a crucial feature—understanding how such views were articulated and normalized is crucial to understanding how they took hold. Yes! > We're developing a responsible access framework that makes models available to researchers for scholarly purp…

It’s as if every researcher in this field is getting high on the small amount of power they have from denying others access to their results. I’ve never been as unimpressed by scientists as I have been in the past five years or so. “We’ve created something so dangerous that we couldn’t possibly live with the moral burden of knowing that the wrong people (which are never us, of course) might get their hands on it, so…

> It’s as if every researcher in this field is getting high on the small amount of power they have from denying others access to their results. I’ve never been as unimpressed by scientists as I have been in the past five years or so.

This is absolutely nothing new. With experimental things, it's non uncommon for a lab to develop a new technique and omit slight but important details to give them a competitive advantage. Similarly in the simulation/modelling space it's been common for years for researchers to not publish their research software. There's been a lot of lobbying on that side by groups such as the Software Sustainability Institute and Research Software Engineer organisations like RSE UK and RSE US, but there's a lot of researchers that just think that they shouldn't have to do it, even when publicly funded.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#223

Post by dr_dshiv »

j-bos wrote: post #210
dr_dshiv wrote: post #197

Everyone learns that the renaissance was sparked by the translation of Ancient Greek works. But few know that the Renaissance was written in Latin — and has barely been translated. Less than 3% of I’m working on a project to change that. Research blog at www.SecondRenaissance.ai — we are starting by scanning and translating thousands of books at the Embassy of the Free Mind in Amsterdam, a UNESCO-recognized rare book…

This ia very cool but should go in a Show HN post as per HN rules. All the best!

Just read the rules again— was something inappropriate? Seemed relevant

Re: History LLMs: Models trained exclusively on pre-1913 texts

#224

Post by zoky »

why-o-why wrote: post #139
zoky wrote: post #101

Earlier quoted context omitted.

Banning Huckleberry Finn from a school district should be grounds for immediate dismissal.

I don't support banning the book, but I think it is hard book to teach because it needs SO much context and a mature audience (lol good luck). Also, there are hundreds of other books from that era that are relevant even from Mark Twain's corpus so being obstinate about that book is a questionable position. I'm ambivalent honestly, but definitely not willing to die on that hill. (I graduated highschool in 1989 from a…

I mean, you gotta read it. I’m not normally a huge fan of the classics; I find Steinbeck dry and tedious, and Hemingway to be self-indulgent and repetitious. Even Twain’s other work isn’t exactly to my taste. But I’ve read Huckleberry Finn three times—in elementary school just for fun, in high school because it was assigned, and I recently listened to it on audiobook—and enjoyed the hell out of each time. Banning it simply because it uses a word that the entire book simply couldn’t exist without is a crime, and does a huge disservice to the very students they are supposedly trying to protect.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#225

Post by tgv »

machinationu wrote: post #202
anotherpaulg wrote: post #100

It would be interesting to see how hard it would be to walk these models towards general relativity and quantum mechanics. Einstein’s paper “On the Electrodynamics of Moving Bodies” with special relativity was published in 1905. His work on general relativity was published 10 years later in 1915. The earliest knowledge cuttoff of these models is 1913, in between the relativity papers. The knowledge cutoffs are also r…

the issue is there is very little text before the internet, so not enough historical tokens to train a really big model

I think not everyone in this thread understands that. Someone wrote "It's a time machine", followed up by "Imagine having a conversation with Aristotle."

Re: History LLMs: Models trained exclusively on pre-1913 texts

#227

Post by dnel »

bikeshaving wrote: post #62
jscyc wrote: post #15

Earlier quoted context omitted.

When you put it that way it reminds me of the Severn/Keats character in the Hyperion Cantos. Far-future AIs reconstruct historical figures from their writings in an attempt to gain philosophical insights.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

Sounds like using Instagram posts to determine what someone really looks like

Re: History LLMs: Models trained exclusively on pre-1913 texts

#228

Post by wongarsu »

ImHereToVote wrote: post #187
seizethecheese wrote: post #119

> Imagine you could interview thousands of educated individuals from 1913—readers of newspapers, novels, and political treatises—about their views on peace, progress, gender roles, or empire. Not just survey them with preset questions, but engage in open-ended dialogue, probe their assumptions, and explore the boundaries of thought in that moment. Hell yeah, sold, let’s go… > We're developing a responsible access fra…

I wonder how much GPU compute you would need to create a public domain version of this. This would be a really valuable for the general public.

To get a single knowledge-cutoff they spent 16.5h wall-clock hours on a cluster of 128 NVIDIA GH200 GPUs (or 2100 GPU-hours), plus some minor amount of time for finetuning. The prerelease_notes.md in the repo is a great description on how one would achieve that

Re: History LLMs: Models trained exclusively on pre-1913 texts

#229

Post by andy99 »

nospice wrote: post #124

I'm surprised you can do this with a relatively modest corpus of text (compared to the petabytes you can vacuum up from modern books, Wikipedia, and random websites). But if it works, that's actually fantastic, because it lets you answer some interesting questions about LLMs being able to make new discoveries or transcend the training set in other ways. Forget relativity: can an LLM trained on this data notice any in…

The chinchilla paper says the “optimal” training data set size is about 20x the number of parameters (in tokens), see table 3: https://arxiv.org/pdf/2203.15556

Here they do 80B tokens for a 4B model.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#230

Post by r0x0r007 »

ffs, to find out what figures from the past thought and how they felt about the world, maybe we read some of their books, we will get the context. Don't prompt or train LLM to do it and consider it the hottest thing since MCP. Besides, what's the point? To teach younger generations a made up perspective of historic figures? Who guarantees the correctness/factuality? We will have students chatting with made up Hitler justifying his actions. So much AI slop everywhere.
Post reply on HN