Viewing profile — ml_hardware

Looks like someone has got DBRX running on an M2 Ultra already: https://x.com/awnihannun/status/1773024954667184196?s=20

Mosaic's MPT models are already supported in GGML: https://github.com/ggerganov/ggml Here's MPT-30B running in 4-bit precision on CPU :) https://twitter.com/abacaj/status/167313344…

The repo for training and finetuning this model is open source here: https://github.com/mosaicml/llm-foundry

Did you actually read the blog? The very first sentence is: > Try out our Stable Diffusion code here! > https://github.com/mosaicml/diffusion-benchmark

I think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't work too well... it de…

I agree! The models will definitely keep getting bigger, and MoEs are a part of that trend, sorry if that wasn’t clear. A pod of gen2-H100s might have 256 GPUs with 40 TB of total …

The 9x speedup is a bit inflated... it's measured at a reference point of ~8k GPUs, on a workload that the A100 cluster is particularly bad at. When measured at smaller #s of GPUs …

Couple points: 1) NVIDIA will likely release a variant of H100 with 2x memory, so we may not even have to wait a generation. They did this for V100-16GB/32GB and A100-40GB/80GB. 2)…

At inference time it will be possible to do 4000 TFLOPS using sparse FP8 :) But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even wi…

Unfortunately it will be hard to investigate properties of large, powerful neural networks without access to their trained weights. And industrial labs that spend millions of dolla…

You may find this blog post useful for thinking about AI scaling: https://www.alignmentforum.org/posts/k2SNji3jXaLGhBeYP/extra... For general tasks like language modeling, we are s…

> "Though the spike seen in the data generates more questions than answers, one thing is clear: A single (albeit large and busy) store’s decision to report a majority of its shopli…

Wow! How are you able to achieve the cost reductions? Is it different hardware, software optimizations, or both? Also does this suggest that OpenAI is charging 6x markups on their …

I was providing costs for the exact instance types that NeuralMagic used in their blog post, if we’re allowed to change that then I can also find cheaper GPU providers. I can agree…

That's true.. in fact, seeing V100 FP16 < T4 FP16 makes me believe you're right, the V100 should be much faster if the tensor cores were being used.

T4 is a gpu :) NVIDIA Tesla T4: https://www.nvidia.com/en-us/data-center/tesla-t4/

My guess is they are using tensor cores as they report FP16 throughput, but they seem to be measuring at batch size 1, which is hugely unfair to the GPUs. For inference workloads y…

I think your math is backwards. The training workload W is the same, and the time to complete it is: W / (256 * speed_v4) = 1.82 W / (4096 * speed_v3) = 0.39 speed_v4 / speed_v3 = …

Haven't seen this mentioned yet, but ResNet-50 is an old model. I would probably expect multiple-percentage-point gains from using a better (or honestly just larger) architecture a…

GPT3 can write correct computer programs. There are many demos online. Stepping back it is just a translation problem e.g. English idea -> Python code, which an ideal language mode…

If you had a language model whose responses were indistinguishable from a human, would you call that understanding? I agree GPT-3 is * just * a language model, but then again, so a…