Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq surpasses 1,200 tokens/sec with Llama 3 8B

twitter.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#11

Post by porphyra »

behnamoh wrote: post #8

They're not responsive to my questions on Twitter, so I'm asking here: When will Groq support a real API (not experimental beta preview)? When will Groq support logprobs?! When will Groq actually tell us what their rate limit is?! Until these aren't answered, many of us can't actually build on Groq. Edit: It seems I'm getting downvoted by Groq employees...

Try asking in the groq discord [0]. Some groq employees are fairly responsive there.

For groqcloud the rate limits are fairly clear [1]. For example, for llama3-8b-8192 you get 30 requests per minute, 14400 per day, and 30000 tokens per minute. That said, it's the beta free tier so it sometimes goes down randomly and the limits may be different once they start charging for it.

I'm not affiliated with groq but I use groqcloud to make some simple chatbots since it's currently free.

[0] https://discord.com/invite/n8KtCjfAug

[1] https://console.groq.com/settings/limits

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#13

Post by BowBun »

porphyra wrote: post #9
windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

Unrelated --- groq wrote an angry blog post complaining about Elon's xAI's grok: https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

Not that angry! I appreciate the tone, though they deserve every right to protect their trademark.

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#15

Post by flutas »

BowBun wrote: post #13
porphyra wrote: post #9

Earlier quoted context omitted.

Unrelated --- groq wrote an angry blog post complaining about Elon's xAI's grok: https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

Not that angry! I appreciate the tone, though they deserve every right to protect their trademark.

> though they deserve every right to protect their trademark.

And they can, Twitter (why everything gets claimed as his personal work I never know) isn't using their trademark.

As they (Groq) themselves have said...

> the difference of one consonant (q, k) only matters to scrabblers and spell checkers

Grok the term has been around since at least 1961.[0] The fact that a company decided to take a common term (especially in the CS field), change one letter and trademark it doesn't mean nobody can use the original spelling at all.

Funnily enough, Groq is trying to claim grok and groq are not associated terms in court filings while trying to bully another company with the same name:

> The word “Groq” essentially did not exist before Ross created it and has no known meaning in any language beyond its intended association with Groq, Inc.

vs that companies reply

> The word “grok” originated in Robert Heinlein’s 1961 novel Stranger in a Strange Land. Merriam Webster defines “grok” as “to understand profoundly and intuitively.” The Oxford English Dictionary defines “grok” as “[t]o understand intuitively or by empathy; to establish rapport with.”

Once Groq realized their trademark didn't include healthcare data, they tried to trademark...the other companies name.

[0]: https://en.wikipedia.org/wiki/Grok

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#16

Post by lxgr »

lxgr wrote: post #7
windowshopping wrote: post #3

Is groq related to Twitter's grok or is that just a very unfortunate naming coincidence?

They seem to be unrelated, but sharing an etymology: https://arxiv.org/abs/2201.02177

Classic HN – downvotes without explanation.

I might well be wrong about the etymology here, but I understand "grokking" to be a term for a phenomenon in training neural networks.

What I'm not sure about is which was there first – AI companies called some version of "grok" or that term.

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#17

Post by flutas »

lxgr wrote: post #16
lxgr wrote: post #7

Earlier quoted context omitted.

They seem to be unrelated, but sharing an etymology: https://arxiv.org/abs/2201.02177

Classic HN – downvotes without explanation. I might well be wrong about the etymology here, but I understand "grokking" to be a term for a phenomenon in training neural networks. What I'm not sure about is which was there first – AI companies called some version of "grok" or that term.

The term grok came from Robert Heinlein’s 1961 novel Stranger in a Strange Land and got picked up by the CS field heavily around the late 60s.

https://en.wikipedia.org/wiki/Grok

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#18

Post by verdverm »

flutas wrote: post #15
BowBun wrote: post #13

Earlier quoted context omitted.

Not that angry! I appreciate the tone, though they deserve every right to protect their trademark.

> though they deserve every right to protect their trademark. And they can, Twitter (why everything gets claimed as his personal work I never know) isn't using their trademark. As they (Groq) themselves have said... > the difference of one consonant (q, k) only matters to scrabblers and spell checkers Grok the term has been around since at least 1961.[0] The fact that a company decided to take a common term (especial…

Trademarks are more nuanced than you are relaying here.

Groq, in arguing that their mark is different from "grok" (at the USPTO) is because one cannot trademark common words. They are applying for plain marks (without font/color/logo) and this is very normal. I went through this with a proper name trademark

In the Groq vs Grok, they are arguing that the average person will confuse the marks (as can be seen in many HN posts about Groq, like this one). Their argument is that Grok should not be given a trademark beforehand due to this potential confusion. They can also take the case to court should the trademark be granted. Given the common confusion, Groq appears to have good standing to make this argument.

To call someone defending their own trademarks "bullying" is inaccurate

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#19

Post by verdverm »

saberience wrote: post #10
andy_xor_andrew wrote: post #6

When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration. My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt. But.…

The issue is that their chips need a huge amount of server blades and there's a big doubt whether this model actually scales. That is, how will Groq handle much larger models with a context of hundreds of thousands or millions of tokens? Right now this would require them to deploy a cluster with thousands of chips, versus 10 chips for say an NVidia system. The other issue they don't mention is power, space, efficienc…

Cerebrus faces similar challenges with their wafer scale chips.

If anything, Google's TPU advancements chart a viable course. I suspect both Groq and Cerebrus will overcome the challenges and offer competitive compute options, depending on the context

Re: Groq surpasses 1,200 tokens/sec with Llama 3 8B

#20

Post by verdverm »

lxgr wrote: post #16
lxgr wrote: post #7

Earlier quoted context omitted.

They seem to be unrelated, but sharing an etymology: https://arxiv.org/abs/2201.02177

Classic HN – downvotes without explanation. I might well be wrong about the etymology here, but I understand "grokking" to be a term for a phenomenon in training neural networks. What I'm not sure about is which was there first – AI companies called some version of "grok" or that term.

From the HN commenting guidelines

> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

Post reply on HN