Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groqchat

chat.groq.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Groqchat

#91

Post by m3kw9 »

jkachmar wrote: post #32
simonw wrote: post #26

Lots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before. What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine". That's explained here: https://groq.com/lpu-inference-engine/ > This is the world’s…

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

Is it fixed to a certain llm architecture like llama2? How does it deal with different architectures like MOE for example

Re: Groqchat

#94

Post by Frummy »

jkachmar wrote: post #71
moneywoes wrote: post #59

Earlier quoted context omitted.

what’s the cost?

right now we’re providing this access to public, anonymous users via this demo chat interface as an alpha test. we’ll be publishing information about API access, and pricing, shortly after the new year.

You should add latex rendering

Re: Groqchat

#95

Post by sandGorgon »

is the TSP a RISC-V on FPGA ? the tweet mentions haskell, which sounds familiar - Bluespec or something.

or is it a completely custom ASIC

Re: Groqchat

#96

Post by [deleted] »

pclmulqdq wrote: post #90
GamerAlias wrote: post #27

In case, it's not blinding obvious to people. Groq are a hardware company that have built chips that are designed around the training and serving of machine models particularly targeted at LLMs. So the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. I actually have a final round interview with a subsidiary of Groq coming up and I'm very undecided as to…

They are putting the whole LLM into SRAM across multiple computing chips, IIRC. That is a very expensive way to go about serving a model, but should give pretty great speed at low batch size.

[deleted]

Re: Groqchat

#97

Post by nomel »

chihuahua wrote: post #84
GamerAlias wrote: post #27

In case, it's not blinding obvious to people. Groq are a hardware company that have built chips that are designed around the training and serving of machine models particularly targeted at LLMs. So the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. I actually have a final round interview with a subsidiary of Groq coming up and I'm very undecided as to…

> the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. But if it was generating high-quality responses, would that not make it go slower?

That would involve using a different model. This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo.

Re: Groqchat

#99

Post by tome »

benchess wrote: post #34

This isn't running on one chip. It's running on 128, or two racks worth of their kit. https://news.ycombinator.com/item?id=38739106 This doesn't mean much without comparing $ or watts of GPU equivalents

Thanks, I need to correct my earlier guess: I believe this demo is running on 9 GroqRacks (576 chips) and I think we may also have an 8 rack version in progress. I can't remember off the top of my head whether this deployment has pipelining of inferences or whether that's work in progress. We've tried a variety of different configurations to improve performance (both latency and throughput), which is possible because of the high level of flexibility and configurability of our architecture and compiler toolchain.

You're right that it is important to compare cost per token also, not just raw speed. Unfortunately I don't have those figures to hand but I think our customer offerings are price competitive with OpenAI's offerings. The biggest takeaway though is that we just don't believe GPU architectures can ever scale to the performance that we can get, at any cost.

Re: Groqchat

#100

Post by tome »

m3kw9 wrote: post #91
jkachmar wrote: post #32

Earlier quoted context omitted.

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

Is it fixed to a certain llm architecture like llama2? How does it deal with different architectures like MOE for example

It's not fixed and our chip wasn't designed with LLMs in mind. It's a general purpose, low latency, high throughput compute fabric. Our compiler toolchain is also general purpose and can compile arbitrary high performance numerical programs without the need for handwritten kernels. Because of the current importance of ML/AI we're focusing on PyTorch and ONNX models as input, but it really could be anything.

We can also deploy speech models like Whisper, for example, or image generation models. I don't know if we have any MOE architectures, but we'll be implementing Mixtral soon for sure!

Post reply on HN