Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

History LLMs: Models trained exclusively on pre-1913 texts

github.com

Post reply on HN
Print view View original

171–180 of 452 posts

Re: History LLMs: Models trained exclusively on pre-1913 texts

#171

Post by NooneAtAll3 »

libraryofbabel wrote: post #14
Heliodex wrote: post #4

The sample responses given are fascinating. It seems more difficult than normal to even tell that they were generated by an LLM, since most of us (terminally online) people have been training our brains' AI-generated text detection on output from models trained with a recent cutoff date. Some of the sample responses seem so unlike anything an LLM would say, obviously due to its apparent beliefs on certain concepts, t…

I used to teach 19th-century history, and the responses definitely sound like a Victorian-era writer. And they of course sound like writing (books and periodicals etc) rather than "chat": as other responders allude to, the fine-tuning or RL process for making them good at conversation was presumably quite different from what is used for most chatbots, and they're leaning very heavily into the pre-training texts. We d…

don't we have parlament transcripts? I remember something about Germany (or maybe even Prussia) developing fast script to preserve 1-to-1 what was said

Re: History LLMs: Models trained exclusively on pre-1913 texts

#172

Post by UltraSane »

bikeshaving wrote: post #62
jscyc wrote: post #15

Earlier quoted context omitted.

When you put it that way it reminds me of the Severn/Keats character in the Hyperion Cantos. Far-future AIs reconstruct historical figures from their writings in an attempt to gain philosophical insights.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

I predict very rich people will pay to have LLMs created based on their personalities.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#173

Post by andai »

AnonymousPlanet wrote: post #92
derrida wrote: post #46

I wonder if you could query some of the ideas of Frege, Peano, Russell and see if it could through questioning get to some of the ideas of Goedel, Church and Turing - and get it to "vibe code" or more like "vibe math" some program in lambda calculus or something. Playing with the science and technical ideas of the time would be amazing, like where you know some later physicist found some exception to a theory or some…

There's an entire subreddit called LLMPhysics dedicated to "vibe physics". It's full of people thinking they are close to the next breakthrough encouraged by sycophantic LLMs while trying to prove various crackpot theories. I'd be careful venturing out into unknown territory together with an LLM. You can easily lure yourself into convincing nonsense with no one to pull you out.

Fully automated toaster-fucker generator!

https://news.ycombinator.com/item?id=25667362

Re: History LLMs: Models trained exclusively on pre-1913 texts

#174

Post by UltraSane »

ghurtado wrote: post #131
bikeshaving wrote: post #62

Earlier quoted context omitted.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

We're literally running out of science fiction topics faster than we can create new ones If I started a list with the things that were comically sci Fi when I was a kid, and are a reality today, I'd be here until next Tuesday.

Not at all, you just need to read different scifi. I suggest Greg Egan and Stephen Baxter and Derek Künsken and The Quantum Thief series

Re: History LLMs: Models trained exclusively on pre-1913 texts

#175

Post by mmooss »

mediaman wrote: post #49
mmooss wrote: post #28

> Imagine you could interview thousands of educated individuals from 1913—readers of newspapers, novels, and political treatises—about their views on peace, progress, gender roles, or empire. I don't mind the experimentation. I'm curious about where someone has found an application of it. What is the value of such a broad, generic viewpoint? What does it represent? What is it evidence of? The answer to both seems to…

This is a regurgitation of the old critique of history: what's it's purpose? What do you use it for? What is its application? One answer is that the study of history helps us understand that what we believe as "obviously correct" views today are as contingent on our current social norms and power structures (and their history) as the "obviously correct" views and beliefs of some point in the past. It's hard for most…

> This is a regurgitation of the old critique of history: what's it's purpose? What do you use it for? What is its application?

Feeling a bit defensive? That is not at all my point; I value history highly and read it regularly. I care about it, thus my questions:

> gives us an interactive compression of the views from a specific point in history without the subsequent coloring by the actual events of history.

What validity does this 'compression' have? What is the definition of a 'compression'? For example, I could create random statistics or verbiage from the data; why would that be any better or worse than this 'compression'?

Interactivity seems to be a negative: It's fun, but it would seem to highly distort the information output from the data, and omits the most valuable parts (unless we luckily stumble across it). I'd much rather have a systematic presentation of the data.

These critiques are not the end of the line; they are step in innovation, which of course raises challenging questions and, if successful, adapts to the problems. But we still need to grapple with them.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#177

Post by paul_h »

d3m0t3p wrote: post #9
_--__--__ wrote: post #6

Earlier quoted context omitted.

The time cutoff probably matters but maybe not as much as the lack of human finetuning from places like Nigeria with somewhat foreign styles of English. I'm not really sure if there is as much of an 'obvious LLM text style' in other languages, it hasn't seemed that way in my limited attempts to speak to LLMs in languages I'm studying.

The model is fined tuned for chat behavior. So the style might be due to - Fine tuning - More Stylised text in the corpus, english evolved a lot in the last century.

Diverged as well as standardized. I did some research into "out of pocket" and how it differs in meaning in UK-English (paying from one's own funds) and American-English (uncontactable) and I recall 1908 being the current thought as to when the divergence happened: 1908 short story by O. Henry titled "Buried Treasure."

Re: History LLMs: Models trained exclusively on pre-1913 texts

#178

Post by LiKao »

9dev wrote: post #150
a1j9o94 wrote: post #140

Earlier quoted context omitted.

Not the person you're responding to, but I think there's a non trivial argument to make that our thoughts are just auto complete. What is the next most likely word based on what you're seeing. Ever watched a movie and guessed the plot? Or read a comment and know where it was going to go by the end? And I know not everyone thinks in a literal stream of words all the time (I do) but I would argue that those people's br…

You, and OP, are taking an analogy way too far. Yes, humans have the mental capability to predict words similar to autocomplete, but obviously this is just one out of a myriad of mental capabilities typical humans have, which work regardless of text. You can predict where a ball will go if you throw it, you can reason about gravity, and so much more. It’s not just apples to oranges, not even apples to boats, it’s app…

Look up predictive coding theory. According to that theory, what our brain does is in fact just autocomplete.

However, what it is doing is layered autocomplete on itself. I.e. one part is trying to predict what the other part will be producing and training itself on this kind of prediction.

What emerges from this layered level of autocompletes is what we call thought.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#179

Post by LiKao »

root_axis wrote: post #153
a1j9o94 wrote: post #140

Earlier quoted context omitted.

Not the person you're responding to, but I think there's a non trivial argument to make that our thoughts are just auto complete. What is the next most likely word based on what you're seeing. Ever watched a movie and guessed the plot? Or read a comment and know where it was going to go by the end? And I know not everyone thinks in a literal stream of words all the time (I do) but I would argue that those people's br…

There's no evidence for it, nor any explanation for why it should be the case from a biological perspective. Tokens are an artifact of computer science that have no reason to exist inside humans. Human minds don't need a discrete dictionary of reality in order to model it. Prior to LLMs, there was never any suggestion that thoughts work like autocomplete, but now people are working backwards from that conclusion base…

There actually was quite a lot of suggestion that thoughts work like autocomplete. A lot of it was just considered niche, e.g. because the mathematical formalisms were beyond what most psychologist or even cognitive scientists would deem usefull.

Predictive coding theory was formalized back around 2010 and traces it roots up to theories by Helmholtz from 1860.

Predictive coding theory postulates that our brains are just very strong prediction machines, with multiple layers of predictive machinery, each predicting the next.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#180

Post by BoredPositron »

NuclearPM wrote: post #75
bikeshaving wrote: post #62

Earlier quoted context omitted.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

[flagged]

I call bullshit because of tone and grammar. Share the chat.
Post reply on HN