Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

161–170 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#161

Post by londons_explore »

mrfinn wrote: post #154
londons_explore wrote: post #138

Earlier quoted context omitted.

They probably trained all 8 experts on the same data. The experts may have become good at different topics, but no human divided up the topics. The output isn't just the best of the 8 experts - it is a blend of the opinions of the experts. Another (usually smaller) neural net decides how to blend together the outputs of the networks, probably on a per-token basis (ie. for each individual word (ie. token), the outputs…

I guess that neural network has to have the capability of identifying the subject and know in every moment which network is the most capable for that subject, otherwise I can't understand how it could possibly evaluate which is the best answer.

Results of this sort of system frequently look almost random to human eyes. For example one expert might be the "capital letter expert", doing a really good job of putting capital letters in the right place in the output.

Re: GPT4 is 8 x 220B params = 1.7T params

#162

Post by joe-user »

matteocontrini wrote: post #112
lannisterstark wrote: post #24

Earlier quoted context omitted.

>his reputation is solid Eh, is it? Not sure if I consider him an authority on anything anymore. https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro... >This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT. is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."

I remember watching a live stream [1] of him going through Twitter engineering articles and trying to reverse engineer the Twitter frontend and backend, and he clearly had absolutely no clue of how anything remotely related to the web works. He was just clicking around compulsively, jumping between stuff randomly without even reading it, while not understanding what he was looking at. I have no idea how he's successf…

> I have no idea how he's successful in tech, judging from what I saw there.

I think judging only from what you saw there is the issue. If you look somewhere like Wikipedia [0], you'll see he was the first person to jailbreak the iPhone, the first person to achieve hypervisor access to the PS3's CPU, he took first place in several notable CTFs (at least one time as a one-person team), he worked on Google's Project Zero team (and created a debugger called QIRA while he was there), creating comma.ai, and the list goes on.

[0]: https://en.wikipedia.org/wiki/George_Hotz

Re: GPT4 is 8 x 220B params = 1.7T params

#163

Post by loudmax »

behnamoh wrote: post #143
espadrine wrote: post #81

Earlier quoted context omitted.

To be fair, this was already a common whisper at the time, so it could be a Chinese whisper effect. Even I thought this on release day: https://news.ycombinator.com/item?id=35165874 What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-sourc…

> The gap with GPT-4 is not that big... Oh it is though. I've tried several OS models and nothing comes even close to GPT-4. Turns out ClosedAI has a moat after all.

AI development is taking place at breathtaking velocity and GPT-4 isn't the end of the road. OpenAI's moat relies on them improving at a faster pace than competitors. I wouldn't take it for granted that OpenAI's lead will remain the same in a year as it is today. Very possibly they could expand their lead. Also possibly, their lead could vanish.

Re: GPT4 is 8 x 220B params = 1.7T params

#164

Post by espadrine »

mark_l_watson wrote: post #158
machdiamonds wrote: post #90

Earlier quoted context omitted.

From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more…

There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements). I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I coul…

I think a lot of people see a larger gap between open-source models and GPT-4 than is present because they test out the 7B models which fit on their machine. Models like T5-FLAN-XXL are very far below the quality one can expect from the best in open-source, and barely usable for CoT or tool use.

Especially for LangChain, I recommend using ≥33B models like Guanaco or WizardLM. Guanaco-65B honestly feels on-par with ChatGPT-3.5. (To be clear, there is a large gap with GPT-4 though.) It is a costly test, although GPTQ (for instance on exllama) help make it affordable.

I haven’t tried Orca since they haven’t released the weights yet, but it doesn’t seem like they have a 33B version.

Re: GPT4 is 8 x 220B params = 1.7T params

#165

Post by blharr »

smif wrote: post #113
jiggywiggy wrote: post #107

I often hear the idea of digital is faster then biology. This seems mostly derived from small math computations. Yet it seems the current form of large language computations is much much slower then our biology. Making it even larger will be necessary to come closer to human levels but the speed? If this is the path to GI, the computational levels need to be very High and very centralized. Are there ways to improve t…

This is probably not the path to GI. First we would need a precise scientific formalism to accurately describe intelligence, which currently does not exist. Second, it may or may not end up being tied to consciousness, and there's a thing called the hard problem of consciousness, that possibly might not be solvable. It might end up being the kind of thing where if you want to accurately model consciousness, you would…

To accurately model consciousness it seems like we'd only need a computer the size of our brains, we'd just need to be as efficient as it.

Re: GPT4 is 8 x 220B params = 1.7T params

#166

Post by cochne »

SheinhardtWigCo wrote: post #10

GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.

No, GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is (24 times X million transistors). It's not wrong to say GPT-4 is 1.7T parameters, your statement is wrong. Parameters can be configured in different ways, just as transistors on a chip can. In the same way that having 24x transistors does not imply 24x performance, having 24x parameters does not imply 24x performance.

Re: GPT4 is 8 x 220B params = 1.7T params

#167

Post by mrtranscendence »

primordialsoup wrote: post #83
swyx wrote: post #75

Earlier quoted context omitted.

(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to yo…

Makes it worse, since you have this understanding and still went with this explanation that it's 1.7 trillion patients

1.7 trillion patients? We're going to need more hospital beds ...

Re: GPT4 is 8 x 220B params = 1.7T params

#168

Post by symbolicAGI »

logicchains wrote: post #159
Tepix wrote: post #100

Earlier quoted context omitted.

220B open source models wouldn't be as useful for most users. You need two RTX 3090 24GB cards already to run inference with a 65B model that is 4bit quantized. Going beyond that (already expensive) hardware is out of reach for the average hobbyist developer.

You could run it quantized to 4 bits on CPU with 256GB ram, which is much cheaper to rent/buy. Sure it might be somewhat slow, but for lots of use cases that doesn't matter.

Benchmarks I've run with a Ryzen 7950x, 128 GB RAM with Nvidia GeForce 3060 12 GB VRAM show a slowdown less than half when not using the GPU, with LLama.cpp as the inference platform and various ggml open source models in the 7B-13B parameter range.

The Ryzen does best with 16 threads, not the 32 it is capable of, which is expected due to it having 16 CPU cores.

Re: GPT4 is 8 x 220B params = 1.7T params

#169

Post by mark_l_watson »

espadrine wrote: post #164
mark_l_watson wrote: post #158

Earlier quoted context omitted.

There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements). I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I coul…

I think a lot of people see a larger gap between open-source models and GPT-4 than is present because they test out the 7B models which fit on their machine. Models like T5-FLAN-XXL are very far below the quality one can expect from the best in open-source, and barely usable for CoT or tool use. Especially for LangChain, I recommend using ≥33B models like Guanaco or WizardLM. Guanaco-65B honestly feels on-par with Ch…

Thanks for your advice! I am trying to write the book examples using a A6000 (48G video memory), but I may have to go higher end. The hourly lease charge on a A6000 is very inexpensive and I wanted the examples to be "almost free" to run. I will see what is required for Guanaco-65B.

Re: GPT4 is 8 x 220B params = 1.7T params

#170

Post by jasonjmcghee »

fnands wrote: post #146
ChatGTP wrote: post #142

Earlier quoted context omitted.

How are. 254090 GPUs supposed to run inside a robot ?

They mean 25 x RTX 4090 GPUs. 4090 is a model number

And to give a bit more context, it's one of the top consumer grade cards available (and has 24GB of RAM). It costs on the order of $1.6k instead of $15-25k of H100.
Post reply on HN