Training for one trillion parameter model backed by Intel and US govt has begun

Post by COAGULOPATH » Sat, Nov 25, 2023, 2:30 AM UTC

chlorion wrote: post #208

haldujai wrote: post #186

Earlier quoted context omitted.

LLMs don’t have insight, these outputs cannot be assumed to be accurate. Due to current hardware limitations it is not feasible to have a 1T parameter model without MoE.

I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI". It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.

Post by rkagerer » Sat, Nov 25, 2023, 3:33 AM UTC

pacificmint wrote: post #227

rkagerer wrote: post #213

Earlier quoted context omitted.

Curious, why do people gravitate to the term "hallucination" (as if it's a person not a tool) instead of just plainly stating that it's wrong?

If I ask it what color an orange it and it says blue, that would be wrong. If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it. Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that w…

Post by _heimdall » Sat, Nov 25, 2023, 4:40 AM UTC

jncfhnb wrote: post #241

_heimdall wrote: post #237

Earlier quoted context omitted.

Monopoly is the byproduct of allowing centralized power, not the natural state. I'm not actually sure how we could narrow down the natural state of humans at this point, but I strongly suspect it wouldn't be based on an assumption that people are willing to give up a growing list of individual freedoms in the name of fear.

Stopping the accumulation of power requires aggressive sacrifice from the less powerful. This isn’t a feature of humans, but basic system dynamics / economics / etc. A group gets more power and leverage that power to gain more power. Inevitable. Coordinated action is the only way to prevent it. And coordinated action is government.

Post by lxgr » Sat, Nov 25, 2023, 5:30 AM UTC

cschmidt wrote: post #160

checkyoursudo wrote: post #113

Earlier quoted context omitted.

Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?

I just asked GPT-4, and it denies being a MoE model. User: Are you an MoE model? ChatGPT: No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach. In a GPT model like mine, the entire mod…

Post by jonplackett » Sat, Nov 25, 2023, 10:24 AM UTC

13of40 wrote: post #167

Majromax wrote: post #41

Earlier quoted context omitted.

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25. Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.…

Nit: The bandwidth of your optic nerve is only about 10 kilobits per second. You think you're seeing in 4K but most of it is synthetic.