Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

251–260 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#251

Post by jonplackett »

PaulDavisThe1st wrote: post #196
jonplackett wrote: post #33

Earlier quoted context omitted.

Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot. But once we are partially trained, training more requires a lot less.

Someone added it up: https://osf.io/preprints/psyarxiv/qzbgx > Large language models show intriguing emergent behaviors, yet they receive around 4-5 orders of magnitude more language data than human children.

_language_ data.

Humans get a lot more input that just language is the point. We get to see language out in the physical world. They have the (really quite difficult) task of inferring all the the real world and its connections, common sense, laws of physics, while locked in a box reading wikipedia.

How capable do you think a child would be if you kept them in a dark box and just spoke to them through the keyhole? (not recommending this experiment)

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#252

Post by sigilis »

rkagerer wrote: post #244
pacificmint wrote: post #227

Earlier quoted context omitted.

If I ask it what color an orange it and it says blue, that would be wrong. If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it. Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that w…

Fair enough and thanks. I felt hallucination was too forgiving a term but I can see how others would rank them the other way around and suppose it works.

The more accurate term I’ve seen floating around is “confabulation”. It’s also a human phenomenon, but it doesn’t bring all the baggage of hallucinations.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#253

Post by ComputerGuru »

feoren wrote: post #219
ComputerGuru wrote: post #152

Earlier quoted context omitted.

This is completely and fundamentally an incorrect approach from start to finish. The human body - and the human mind - do have electrical and logic components but they are absolutely not digital logic. We do not see in “pixels”. The human mind is an analog process. Analog computing is insanely powerful and exponentially more efficient (time, energy, bandwidth) than digital computing but it is ridiculously hard to pac…

You missed the point of the exercise. Of course it's extremely difficult to compare the two, but the question was: do humans get nearly as much training data as LLMs do? This analysis is good enough to say "actually humans receive much more raw input data than LLMs during their 'training period'." You're concerned with what the brain is doing with all that data compared to LLMs, but that's not the point of the exerci…

No, you can’t even compare that because the information isn’t packetized. Again, we don’t see in pixels so you can’t just consider sensory input in that manner.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#254

Post by paulddraper »

madsbuch wrote: post #96
_heimdall wrote: post #44

I'm sure the government's mission is also to develop an AGI that benefits us all.

This is a particular interesting part of American culture: The sentiment is that it is problematic when the government develops these technologies, but it is completely OK to let private entities develop them.

The government is far more powerful than any private entity

And there's no unsubscribe button

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#255

Post by chlorion »

COAGULOPATH wrote: post #242
chlorion wrote: post #208

Earlier quoted context omitted.

I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI". It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.

Yes, it has an invisible "shadow prompt" that gets sent to it when you start a session. It looks something like this: https://www.reddit.com/r/ChatGPT/comments/zo9of4/comment/j0n...

That makes sense, there has to be some level of human written responses for specific things, like when you ask it about a controversial topic that it refuses to answer, it gives you some canned response about how it won't answer that, and that very obviously isn't what language models do naturally. For these cases there has to be some sort of pre-programmed human generated reply!

It's interesting that other people don't seem to agree with the idea that it has some pre-programmed responses, I am curious as to what they think is going on here.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#256

Post by barnabee »

Zpalmtree wrote: post #133
barnabee wrote: post #18

Earlier quoted context omitted.

This reads like a combination of pessimism and nostalgia has allowed you to cherry pick the worst of now and the best of the past and conince yourself it is net bad. I see all of these negatives, and agree it is clear there is much that is bad in the world (there always has been) and much to be improved (ditto), including new problems created by science, technology, and human greed & selfishness. But I also see many…

The positive effects are also the results of capitalism

Yes, to a large extent they are.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#257

Post by sunnybeetroot »

2OEH8eoCRo0 wrote: post #212
Jensson wrote: post #209

Earlier quoted context omitted.

Why not? Would be cool with some new open source models.

I agree but I don't think our adversaries should get a freebie that's trained on our scientific data at a national laboratory. Which makes me wonder. I'm not sure this applies here but say you train a model on classified information, is the model/weights then classified?

If a model is trained on illegally obtained copyrighted material, are the images it produces stolen?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#258

Post by jncfhnb »

_heimdall wrote: post #246
jncfhnb wrote: post #241

Earlier quoted context omitted.

Stopping the accumulation of power requires aggressive sacrifice from the less powerful. This isn’t a feature of humans, but basic system dynamics / economics / etc. A group gets more power and leverage that power to gain more power. Inevitable. Coordinated action is the only way to prevent it. And coordinated action is government.

It sounds like you're describing coordinated action as both the cause of and solution to the same problem. Stopping this accumulation of power takes very little, its undoing the accumulation of power that is costly.

Stopping accumulation of power is very hard when you have no accumulated power. I’m not sure how you could possibly believe otherwise unless you’re one of those believers in magical harmonious anarchy

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#259

Post by lporto »

WhitneyLand wrote: post #243

Is anything known about what extent if any non-public domain books are used for LLM’s? One example is the Google books project made digital quite a few texts, but I’ve never heard if Google considers these fair game to train on for Bard. Most of the copyright discussions I’ve seen have been around images and code but not much about books. Seems to become more relevant as things scale up as indicated by this article.

>we found 72,508 ebook titles (including 83 from Stanford University Press) that were pirated and then widely used to train LLMs despite the protections of copyright law

https://aicopyright.substack.com/p/the-books-used-to-train-l...

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#260

Post by upsidesinclude »

Haha, this is funny because everyone is talking about this as if it is designed to be like the LLMs we have access to.

The training parameters will be the databases of info scooped up and integrated into profiles of every person and their entire digital footprint, queriable and responsive to direct questioning

Post reply on HN