GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.
I think frequency is not additive like the parameters. If we are looking for analogy in compute power, then FLOPs is a better analogy to parameters.
GPT4 is 8 x 220B params = 1.7T params
71–80 of 215 posts
Re: GPT4 is 8 x 220B params = 1.7T params
#72Earlier quoted context omitted.
>his reputation is solid Eh, is it? Not sure if I consider him an authority on anything anymore. https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro... >This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT. is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."
Well he picked the right problem at the time which was (and still is!) search. I was really looking forward to some progress on this and then he deleted tweets and dropped off the system. I would love to see a podcast of him talking about that & fyi @realGeorgeHotz is back on it now to promote Tiny grad/box/corp.
As someone who has spent 20+ years working in large companies you can spot developers like him a mile away i.e. the sort of behaviour you see with skilled but arrogant grads/interns.
The right approach when you're new is to quietly pick a simple problem away from the core services where you can learn the processes and polices needed to get something into Production. More so when you're in a company that is undergoing a brain drain.
Then you can progressively move to solving more demanding and critical issues.
Re: GPT4 is 8 x 220B params = 1.7T params
#73Earlier quoted context omitted.
This wasn’t very impressive: https://www.pcmag.com/news/hacker-george-hotz-resigns-from-t... Spent two weeks trying to find someone to build a faceted search UI and then quit.
His biggest issue was trying to convince the remaining engineers and Elon that a refactor was more promising than continuing to build on the layers of spaghetti code that runs twitter. I don't disagree with him, it's just clear that he didn't understand how dire the financial situation was and that even a progressive refactor starting with the most basic features would take considerable engineering hours and money.
As the saying goes, if you don’t have time to do it right, you’d better have time to do it twice.
Re: GPT4 is 8 x 220B params = 1.7T params
#74I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
It’s fine to say you don’t believe him, it’s fine to say he’s wrong, but if you want to claim he doesn’t have inside information, I would expect a standard of evidence as good as that you would expect for Geohot’s claim. So tell me: how do you know the Geohot doesn’t have inside information?
It's up to Geohot to prove his claim.
Althoug he may have insider information, he isn't in a position known to have inside information. It might be more correct to phrase it that way but we all got the meaning.
Re: GPT4 is 8 x 220B params = 1.7T params
#75weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be th…
(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to you to do this "job"
ultimately my goal is to Learn in Public and demonstrate to experts that spending their time teaching/sharing with me is a good use of time because i will augment/amplify/simplify their message.
(pls give the full podcast a listen/read/watch, George went deep on tinygrad/tinybox/tinycorp and there's lots there he IS the authority on, and people are overly fixated on the GPT4 rumor https://www.latent.space/p/geohot#details )
Re: GPT4 is 8 x 220B params = 1.7T params
#76Interesting. And makes sense. Eg I could see one of the eight being very close focussed and trained on GitHub like data. Could help it stay on tasks too
i'd love any and all informed speculation on what the 8 models could be. it seems fascinating that we don't know.
Re: GPT4 is 8 x 220B params = 1.7T params
#77Earlier quoted context omitted.
I think deciding to get away from the Musk/Twitter debacle as soon as you realize how bad it is isn't necessarily a bad thing..
How is it a debacle and how is it bad?
Because Twitter is trending towards bankruptcy.
a) It is valued at about a 1/4 of what it was purchased at.
b) Twitter Blue has generated an irrelevant amount of revenue and churn is increasing [1].
c) Roadmap looks poor. Video is a terrible direction where only Google, Amazon, TikTok etc have been able to make the numbers work and that's because it is subsidised through other revenue sources. Payments is DOA given Twitter's inability to comply with existing regulations let alone how difficult KYC/AML is to manage.
d) Regulatory and legal risk increases by the day. Lawsuits continue to pile up and EU/FTC are hovering around as Twitter is not in compliance with previous agreements.
e) Brand safety continues to be a huge challenge that isn't solvable without effectively going back to what Twitter was previously.
f) BlueSky and Instagram are both releasing competitor apps to the broader public in the coming months. The market simply won't sustain this many text-based social media apps.
Re: GPT4 is 8 x 220B params = 1.7T params
#78Earlier quoted context omitted.
https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh20... I think they _are_ going pretty extreme now.
Offtopic, but as a VR gamer that article just made me very sad. I was really hoping to see NVidia produce some decent cards in the near future, but looks like their main revenue is really going to be gargantuan number-crunchers. They'll likely only keep increasing the VRAM of gaming cards by arbitrarily-small numbers once every few years :-(
Yes, the future of VR gaming looks closer to the Sony Playstation or even the Apple Vision than NVIDIA's products.
Re: GPT4 is 8 x 220B params = 1.7T params
#79Earlier quoted context omitted.
It's really interesting that human organism requires so much computational power to support live.
I think it's even more interesting that the required amount of energy to do that high computational work isn't that high. Evolution has been working on it for a long time, and some things are really inefficient but overall it does an OK job at making squishy machines.
I had a good chuckle at "squishy machines". That's a really interesting way to think about it. It makes me wonder if, some day, we will be able to build "squishy machines" of our own, capable of outperforming silicon while using a tiny fraction of the energy.
Re: GPT4 is 8 x 220B params = 1.7T params
#80I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
That's luckily not relevant here. Multiple sources have independently confirmed the same rumors. This means that there is a significant probability of it being true.