Viewing profile — z4y5f3

They will release the weights by 7/27 along with support in vLLM. Stop second guessing. Source: their blog post https://mp.weixin.qq.com/s/V4xhEIy8xDXSMDPrPkmUAQ

They will release the full weights by 7/27 along with support in vLLM. Source: their release blog on WeChat. https://mp.weixin.qq.com/s/V4xhEIy8xDXSMDPrPkmUAQ

NVIDIA is likely citing 1 PFlops at FP 4 sparse (they did this for GB200), so that is 128 TFlops BF16 dense, or 2/3 of what RTX 4090 is capable of. I would put the memory bandwidth…

Yep I have seen this paper before, and thank you for linking it here for reference. My personal opinion is that compared to single epoch scaling laws, we still need more evidence a…

What they missed is that current scaling laws (OpenAI, Deepmind Chinchilla) are based on the assumption that the model is trained for one epoch. This essentially means that in orde…

My experience is that < 500M models are pretty useful when fine-tuned on traditional NLP tasks, such as text classification and sentence/token level labeling. A modern LM with a 32…

Checked your numbers in another thread - excellent breakdown. Thanks for the clarification. I did not read the original anandtech post so I did not realize the 512GBps already refe…

NVLink advertises combined bandwidth in both direction, so the 1800 GBps NVLink on Blackwell is actually 900 GBps for everyone else. PCIe can also do multi-node direct transfer via…

Depends. NVLink advertises bidirectional bandwidth, whereas PCIe and standard networking calculate bandwidth in a single direction. So a 1800 GBps NVLink is actually 900 GBps in PC…

Unfortunately GZIP won't beat LLMs for text classification. The research you cited is just poorly done science that has been widely debunked. The original paper compared top-2 accu…