Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groqchat

chat.groq.com

Post reply on HN
Print view View original

21–30 of 131 posts

Re: Groqchat

#23

Post by andygeorge »

badFEengineer wrote: post #3

This was surprisingly fast, 276.27 T/s (although Llama 2 70B is noticeably worse than GPT-4 turbo). I'm actually curious if there's good benchmarks for inference tokens per second- I imagine it's a bit different for throughput vs. single inference optimization, but curious if there's an analysis somewhere on this edit: I re-ran the same prompt on perplexity llama-2-70b and getting 59 tokens per sec there

fast but wrong/gibberish

Re: Groqchat

#24

Post by coder543 »

alex_young wrote: post #20

[flagged]

It’s just running bog standard Llama2-70B by all appearances.

I don’t know why so many people here are interested in the outputs. The whole point of this demo is that the company is trying to show off how fast their hardware could host one of your models, not the model itself.

Re: Groqchat

#25

Post by razorguymania »

andygeorge wrote: post #23
badFEengineer wrote: post #3

This was surprisingly fast, 276.27 T/s (although Llama 2 70B is noticeably worse than GPT-4 turbo). I'm actually curious if there's good benchmarks for inference tokens per second- I imagine it's a bit different for throughput vs. single inference optimization, but curious if there's an analysis somewhere on this edit: I re-ran the same prompt on perplexity llama-2-70b and getting 59 tokens per sec there

fast but wrong/gibberish

Its using vanilla llama-2 from Meta with no fine tuning. The point here is the speed and responsiveness of the underlying HW and SW.

Re: Groqchat

#26

Post by simonw »

Lots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before.

What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine".

That's explained here: https://groq.com/lpu-inference-engine/

> This is the world’s first Language Processing Unit™ Inference Engine, purpose-built for inference performance and precision. How performant? Today, we are running Llama-2 70B at over 300 tokens per second per user.

I think the LPU is a custom hardware chip, though the page talking about it doesn't make that as clear as it could.

https://groq.com/products/ makes it a bit more clear - there's a custom chip, "GroqChip™ Processor".

Re: Groqchat

#27

Post by GamerAlias »

In case, it's not blinding obvious to people. Groq are a hardware company that have built chips that are designed around the training and serving of machine models particularly targeted at LLMs. So the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second.

I actually have a final round interview with a subsidiary of Groq coming up and I'm very undecided as to whether to pursue it so this felt extraordinarily serendipitous to me. Food for thought shown here

Post reply on HN