Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

History LLMs: Models trained exclusively on pre-1913 texts

github.com

Post reply on HN
Print view View original

131–140 of 452 posts

Re: History LLMs: Models trained exclusively on pre-1913 texts

#131

Post by ghurtado »

bikeshaving wrote: post #62
jscyc wrote: post #15

Earlier quoted context omitted.

When you put it that way it reminds me of the Severn/Keats character in the Hyperion Cantos. Far-future AIs reconstruct historical figures from their writings in an attempt to gain philosophical insights.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

We're literally running out of science fiction topics faster than we can create new ones

If I started a list with the things that were comically sci Fi when I was a kid, and are a reality today, I'd be here until next Tuesday.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#132

Post by root_axis »

A4ET8a8uTh0_v2 wrote: post #110
root_axis wrote: post #105

Earlier quoted context omitted.

> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By…

But.. and I am not asking it for giggles, does it mean humans are giant autocomplete machines?

Not at all. Why would it?

Re: History LLMs: Models trained exclusively on pre-1913 texts

#133

Post by NiloCK »

root_axis wrote: post #105
libraryofbabel wrote: post #63

Earlier quoted context omitted.

This is the 2023 take on LLMs. It still gets repeated a lot. But it doesn’t really hold up anymore - it’s more complicated than that. Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. Sure, LLMs do not think like humans and they may not have human-level creativity. Sometimes…

> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By…

First: a selection mechanism is just a selection mechanism, and it shouldn't confuse the observation of an emergent, tangential capabilities.

Probably you believe that humans have something called intelligence, but the pressure that produced it - the likelihood of specific genetic material to replicate - it is much more tangential to intelligence than next-token-prediction.

I doubt many alien civilizations would look at us and say "not intelligent - they're just genetic information replication on steroids".

Second: modern models also under go a ton of post-training now. RLHF, mechanized fine-tuning on specific use cases, etc etc. It's just not correct that token-prediction loss function is "the whole thing".

Re: History LLMs: Models trained exclusively on pre-1913 texts

#134

Post by ghurtado »

NuclearPM wrote: post #75
bikeshaving wrote: post #62

Earlier quoted context omitted.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

[flagged]

Depending on which prompt you used, and the training cutoff, this could be anywhere from completely unremarkable to somewhat interesting.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#135

Post by A4ET8a8uTh0_v2 »

root_axis wrote: post #132
A4ET8a8uTh0_v2 wrote: post #110

Earlier quoted context omitted.

But.. and I am not asking it for giggles, does it mean humans are giant autocomplete machines?

Not at all. Why would it?

Call it a.. thought experiment about the question of scale.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#136

Post by idiotsecant »

bikeshaving wrote: post #62
jscyc wrote: post #15

Earlier quoted context omitted.

When you put it that way it reminds me of the Severn/Keats character in the Hyperion Cantos. Far-future AIs reconstruct historical figures from their writings in an attempt to gain philosophical insights.

This isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ

Zero percent chance this is anything other than laughably bad. The fact that they're trotting it out in front of the press like a double spaced book report only reinforces this theory. It's a transparent attempt by someone at the CIA to be able to say they're using AI in a meeting with their bosses.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#138

Post by root_axis »

dash2 wrote: post #118
root_axis wrote: post #105

Earlier quoted context omitted.

> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By…

This would be true if all training were based on sentence completion. But training involving RLHF and RLAIF is increasingly important, isn't it?

Reinforcement learning is a technique for adjusting weights, but it does not alter the architecture of the model. No matter how much RL you do, you still retain all the fundamental limitations of next-token prediction (e.g. context exhaustion, hallucinations, prompt injection vulnerability etc)

Re: History LLMs: Models trained exclusively on pre-1913 texts

#139

Post by why-o-why »

zoky wrote: post #101
aidenn0 wrote: post #99

Earlier quoted context omitted.

> And as a result, no one is banning these books (except conservatives that want to retcon american history). My (very liberal) local school district banned English teachers from teaching any book that contained the n-word, even at a high-school level, and even when the author was a black person talking about real events that happened to them. FWIW, this was after complaints involving Of Mice and Men being on the cur…

Banning Huckleberry Finn from a school district should be grounds for immediate dismissal.

I don't support banning the book, but I think it is hard book to teach because it needs SO much context and a mature audience (lol good luck). Also, there are hundreds of other books from that era that are relevant even from Mark Twain's corpus so being obstinate about that book is a questionable position. I'm ambivalent honestly, but definitely not willing to die on that hill. (I graduated highschool in 1989 from a middle class suburb, we never read it.)

Re: History LLMs: Models trained exclusively on pre-1913 texts

#140

Post by a1j9o94 »

root_axis wrote: post #137
A4ET8a8uTh0_v2 wrote: post #135

Earlier quoted context omitted.

Call it a.. thought experiment about the question of scale.

I'm not exactly sure what you mean. Could you please elaborate further?

Not the person you're responding to, but I think there's a non trivial argument to make that our thoughts are just auto complete. What is the next most likely word based on what you're seeing. Ever watched a movie and guessed the plot? Or read a comment and know where it was going to go by the end?

And I know not everyone thinks in a literal stream of words all the time (I do) but I would argue that those people's brains are just using a different "token"

Post reply on HN