Groq surpasses 1,200 tokens/sec with Llama 3 8B • Hacker News Board

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#31

Post by frozenport » Sun, Jun 09, 2024, 8:43 PM UTC

snhbsqub wrote: post #28

frozenport wrote: post #25

Earlier quoted context omitted.

Samba is on gen 4 silicon and still lagging, somebody over there is doing something wrong

How are they lagging? They are running faster than anyone else at full precision and with many many fewer chips than Groq. Groq is not real.

Well I've been using the groq public api, and its approx. the rates claimed.

Economics and costs are hard to predict. For example, Groq is not using HBM chips. So probably the cards are a lot easier to source.

Its not clear what the capacity of these systems are in terms of total users, or even tokens per second. Then you factor in cost. Then you realize all vendors will match a competitors pricing. Then you realize Groq doesn't sell chips.

¯\_(ツ)_/¯

The only thing you have is the public API to benchmark against: https://artificialanalysis.ai/

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#32

Post by snhbsqub » Wed, Jun 12, 2024, 3:35 PM UTC

frozenport wrote: post #31

snhbsqub wrote: post #28

Earlier quoted context omitted.

How are they lagging? They are running faster than anyone else at full precision and with many many fewer chips than Groq. Groq is not real.

Well I've been using the groq public api, and its approx. the rates claimed. Economics and costs are hard to predict. For example, Groq is not using HBM chips. So probably the cards are a lot easier to source. Its not clear what the capacity of these systems are in terms of total users, or even tokens per second. Then you factor in cost. Then you realize all vendors will match a competitors pricing. Then you realize…

- Groq has exactly 0 dollars in revenue - Groq requires 576 chips to run a single model - Groq can do low latency inference, but can't handle batches, and can't run a diversity of different models on each deployment - Groq quantizes the models, significantly affecting quality to get more speed (and don't communicate this to end users, which is very deceptive) - Groq can only run inference, cannot train on their systems

- SambaNova has real revenue from big customers - SambaNova can run any model on a single node at the speed Groq requires - SambaNova can do low latency inference just like Groq, but can also run large batches and host hundreds of models on a single deployment - SambaNova does not quantize models unless explicitly stated - SambaNova can run training at perf competitive with Nvidia, as well as fastest inference in the world at full precision

It really isn't a competition. Groq has done great as garnering hype in recent months, but it is a house of cards.

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#33

Post by frozenport » Fri, Jun 14, 2024, 2:28 AM UTC

snhbsqub wrote: post #32

frozenport wrote: post #31

Earlier quoted context omitted.

Well I've been using the groq public api, and its approx. the rates claimed. Economics and costs are hard to predict. For example, Groq is not using HBM chips. So probably the cards are a lot easier to source. Its not clear what the capacity of these systems are in terms of total users, or even tokens per second. Then you factor in cost. Then you realize all vendors will match a competitors pricing. Then you realize…

- Groq has exactly 0 dollars in revenue - Groq requires 576 chips to run a single model - Groq can do low latency inference, but can't handle batches, and can't run a diversity of different models on each deployment - Groq quantizes the models, significantly affecting quality to get more speed (and don't communicate this to end users, which is very deceptive) - Groq can only run inference, cannot train on their syste…

I think semi analysis commented that they have pipelines instead of batches[1].

So every clock cycle you're doing useful work rather than loading up people into batches. And thats why the arch will probably win for inference, for training you're basically competing with software eco system and silicon density. AKA NVIDIA can give TSMC more money to get more ALUs on the die.

I think other places have attempted dataflow (FPGA etc) but they all basically had buffers (due to non-determinism in networks stack and even ram). SambaNova seems indistinguishable from an FPGA with a few clock cycles difference. I think they blew their shot with a Series D ($600 million???) where they made more of the same old. Maybe Intel will buy them to augment Altera? Looks like chasing parity with existing strategies.

I buy the Groq hype because its something different, certainly the public demo helped. HN is about the future.

[1] https://www.semianalysis.com/p/groq-inference-tokenomics-spe...