31–40 of 215 posts
I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
He doesn't strike me as the type of person to lie (except when trolling). His reputation is solid enough that I'm sure he's had discussions with people in the space.
Spent two weeks trying to find someone to build a faceted search UI and then quit.
GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.
Earlier quoted context omitted.
He doesn't strike me as the type of person to lie (except when trolling). His reputation is solid enough that I'm sure he's had discussions with people in the space.
>his reputation is solid Eh, is it? Not sure if I consider him an authority on anything anymore. https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro... >This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT. is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."
It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now.
Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly"
There is also this linked tweet https://twitter.com/LiamFedus/status/1536791574612303872 - "They are all related to Switch-Transformers and MoE. Of the 3 people on Twitter, 2 joined OpenAI. Could be related, could be unrelated"
Which links to this tweet: "Today we're releasing all Switch Transformer models in T5X/JAX, including the 1.6T param Switch-C and the 395B param Switch-XXL models. Pleased to have these open-sourced!"
Anyway... remember not to just read the headlines, they can be misleading.
Earlier quoted context omitted.
It doesn't fit in VRAM.
I’ve been a bit surprised that Nvidia hasn’t gone to extreme lengths to fit 1tb of memory on a card just for this reason.
Cost aside, HMB dies themselves aren't getting significantly denser anytime soon, and there just simply isn't enough package space with current manufacturing methods to pack a significantly increased number of dies on the gpu.
So I suspect the major hardware jumps will continue to be with NVLink/NVSwitch. Nvlink 4 + NVSwitch 3 actually already allows for up 256x GPUs https://resources.nvidia.com/en-us-grace-cpu/nvidia-grace-ho... ; increased numbers of links will let ever increasing numbers of GPUs pool with sufficient bandwidth for inference on larger models.
As already mentioned, see this HN post about the GH200 https://news.ycombinator.com/item?id=36133226, which has some further discussion about the cutting edge of bandwidth for Nvidia DGX and Google TPU pods.
Is this still orders of magnitude smaller than a human brain? How many? Based on current human neurons/synapses knowledge?
2 orders of magnitude smaller, assuming 100T synaptic connections in the human brain.
Or the models are all trained on the same corpus, but just queried with different parameters?
Is this functionally the same as beam search?
Do they select the best output on a token-by-token basis, or do they let each model stream to completion and then pick the best final output?
Earlier quoted context omitted.
He doesn't strike me as the type of person to lie (except when trolling). His reputation is solid enough that I'm sure he's had discussions with people in the space.
This wasn’t very impressive: https://www.pcmag.com/news/hacker-george-hotz-resigns-from-t... Spent two weeks trying to find someone to build a faceted search UI and then quit.
Is this still orders of magnitude smaller than a human brain? How many? Based on current human neurons/synapses knowledge?
We really have no idea how to directly compare the two. Also, vast portions of the human brain are dedicated to the visual cortex, smelling, breathing, muscle control... things which have value to us but which don't contribute to knowledge work when evaluating how many parameters it would take to replace human knowledge work.