Groq runs Mixtral 8x7B-32k with 500 T/s

Post by moffkalast » Tue, Feb 20, 2024, 3:03 PM UTC

tome wrote: post #450

moffkalast wrote: post #432

Earlier quoted context omitted.

I think NVidia is listing max throughput in terms of batching, so e.g. 50 tok/s for 10 different prompts at the same time. Groq LPUs definitely outerform an H100 in raw speed. But fundamentally it's a system that only has 10x the speed for 500x the price, made by a company that runs a blockchain and is trying to heavily market what were intended to be crypto mining chips for LLM inference. It's really quite a funny c…

Groq does not run a blockchain and our chips were never intended for crypto mining.

Post by tome » Tue, Feb 20, 2024, 3:15 PM UTC

moffkalast wrote: post #451

tome wrote: post #450

Earlier quoted context omitted.

Groq does not run a blockchain and our chips were never intended for crypto mining.

https://www.livecoinwatch.com/price/GroqAI-GROQ I suppose that's someone else then? If that's true, then with this and Elon's Grok it's surprising the US Patent office hasn't taken your trademark away yet for not adequately defending it from infringement.

Post by pclmulqdq » Tue, Feb 20, 2024, 4:08 PM UTC

Temporary_31337 wrote: post #389

pclmulqdq wrote: post #299

Earlier quoted context omitted.

You can control when those happen, too.

not without affecting performance though? If you delay refreshes, this lowers performance as far as I remember...

Post by modeless » Tue, Feb 20, 2024, 5:54 PM UTC

noir_lord wrote: post #431

modeless wrote: post #312

Earlier quoted context omitted.

All that matters is the cost. Their price is cheap, so the real question is whether they are subsidizing the cost to achieve that price or not.

> All that matters is the cost. Not really, sustainability matters, if they are the only game in town, you want to know that game isn't going to end suddenly when their runway turns into a brick wall.

Post by johndough » Tue, Feb 20, 2024, 7:34 PM UTC

jart wrote: post #379

johndough wrote: post #303

Earlier quoted context omitted.

You can ask your website: "What is the computational complexity of self-attention with respect to input sequence length?" It'll answer something along the lines of self-attention being O(n^2) (where n is the sequence length) because you have to compute an attention matrix of size n^2. There are other attention mechanisms with better computational complexity, but they usually result in worse large language models. To…

Changing the way transformer models works is orthogonal to gaining good performance on Mistral. Groq did great work reducing the latency considerably of generating tokens during inference. But I wouldn't be surprised if they etched the A matrix weights in some kind of fast ROM, used expensive SRAM for the the skinny B matrix, and sent everything else that didn't fit to good old fashioned hardware. That's great for ge…

    import urllib.request, json, math

    for i in range(1, 20):
        url = f"https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/resolve/main/model-{i:05d}-of-00019.safetensors?download=true"

        with urllib.request.urlopen(url) as r:
            header_size = int.from_bytes(r.read(8), byteorder="little")
            header = json.loads(r.read(header_size).decode("utf-8"))
            for name, value in header.items():
                if name.endswith(".weight"):
                    shape = value["shape"]
                    mb = math.prod(shape) * 2e-6
                    print(mb, "MB for", shape, name)