Groq surpasses 1,200 tokens/sec with Llama 3 8B

#1

Post by YourCupOTea » Thu, May 30, 2024, 6:52 PM UTC

Groq surpasses 1,200 tokens/sec with Llama 3 8B
twitter.com

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#2

Post by LorenDB » Thu, May 30, 2024, 6:57 PM UTC

Groq is an insane company. SambaNova (discussed yesterday[0]) is also very promising. However, what I really want to see is local AI accelerator chips a la Tenstorrent Grayskull that can boost local generation to hundreds of tokens per second while being more efficient than GPUs.

[0]: https://news.ycombinator.com/item?id=40508797

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#3

Post by windowshopping » Thu, May 30, 2024, 7:07 PM UTC

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#4

Post by Me1000 » Thu, May 30, 2024, 7:08 PM UTC

windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Completely unrelated.

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#5

Post by verdverm » Thu, May 30, 2024, 7:09 PM UTC

windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Different companies and efforts

Groq - mainly hardware, the LPU (https://wow.groq.com/lpu-inference-engine/)

Grok - Elon's de jour AI endeavor

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#6

Post by andy_xor_andrew » Thu, May 30, 2024, 7:14 PM UTC

When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration.

My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt.

But... then I tried their demo, and... yeah, it's that good. Such an amazing company of talented individuals.

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#7

Post by lxgr » Thu, May 30, 2024, 7:16 PM UTC

windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

They seem to be unrelated, but sharing an etymology: https://arxiv.org/abs/2201.02177

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#8

Post by behnamoh » Thu, May 30, 2024, 7:18 PM UTC

They're not responsive to my questions on Twitter, so I'm asking here:

    When will Groq support a real API (not experimental beta preview)?

    When will Groq support logprobs?!

    When will Groq actually tell us what their rate limit is?!

Until these aren't answered, many of us can't actually build on Groq.

Edit: It seems I'm getting downvoted by Groq employees...

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#9

Post by porphyra » Thu, May 30, 2024, 7:22 PM UTC

windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Unrelated --- groq wrote an angry blog post complaining about Elon's xAI's grok: https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#10

Post by saberience » Thu, May 30, 2024, 7:24 PM UTC

andy_xor_andrew wrote: post #6

When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration. My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt. But.…

The issue is that their chips need a huge amount of server blades and there's a big doubt whether this model actually scales. That is, how will Groq handle much larger models with a context of hundreds of thousands or millions of tokens? Right now this would require them to deploy a cluster with thousands of chips, versus 10 chips for say an NVidia system.

The other issue they don't mention is power, space, efficiency etc. We want to run larger models with less power, fewer server blades, at lower cost. Not use more server blades, more chips, more power, etc.