Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groqchat

chat.groq.com

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Groqchat

#101

Post by tome »

cicce19 wrote: post #83
jkachmar wrote: post #32

Earlier quoted context omitted.

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

Will you be selling individual cards? Are you looking for use cases in the healthcare vertical (noticed its not on your current list)? Working in the medical imaging space and could use this tech as part of the offering. Reach out at 16bit.ai

You can buy individual cards. For example Bittware is a reseller: https://www.bittware.com/products/groq/

But it might be best if you just contact us to explain your needs: https://groq.com/contact/

I can also pass your details on to our sales team.

Re: Groqchat

#102

Post by tome »

m1sta_ wrote: post #79
jkachmar wrote: post #32

Earlier quoted context omitted.

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

How easy is it for companies to setup private local servers using Grow hardware (cost and complexity). I've got money. I want throughout.

We've built and deployed racks at a number of organizations. Can you write a message to sales explaining your needs? https://groq.com/contact/

Or if you give me your contact details I can pass them on.

Re: Groqchat

#103

Post by tome »

mlazos wrote: post #78
jkachmar wrote: post #32

Earlier quoted context omitted.

this is running on custom hardware, if you’re curious about the underlying architecture check the publication below. https://groq.com/wp-content/uploads/2023/05/GroqISCAPaper202... EDIT: i work at Groq, but i’m commenting in a personal capacity. happy to answer clarifying questions or forward them along to folks who can :)

How many chips are used for this demo? Do they have dram? I remember the earlier versions did not have dram. Are they also used for training or just inference?

I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total.

Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!

Re: Groqchat

#104

Post by tome »

jkachmar wrote: post #71
moneywoes wrote: post #59

Earlier quoted context omitted.

what’s the cost?

right now we’re providing this access to public, anonymous users via this demo chat interface as an alpha test. we’ll be publishing information about API access, and pricing, shortly after the new year.

Yup, we will be price competitive with OpenAI, and much faster!

Re: Groqchat

#105

Post by tome »

laborcontract wrote: post #58
simonw wrote: post #26

Lots of comments talking about the model itself. This is Llama 2 70B, a model that has been around for a while now, so we're not seeing anything in terms of model quality (or model flaws) we haven't seen before. What's interesting about this demo is the speed at which it is running, which demonstrates the "Groq LPU™ Inference Engine". That's explained here: https://groq.com/lpu-inference-engine/ > This is the world’s…

This is really impressive. For reference, inference for llama 70b on together’s api generates text at roughly 60 tokens/second. I can’t find any information about an api, though I’m guessing that the costs are eye watering. If they offered a Mixtral endpoint that did 300-400 tokens per second at a reasonable cost, I can’t imagine ever using another provider.

We don't have an API in public availability yet but that's coming soon in the new year. We will be price competitive with OpenAI but much faster. Deploying Mixtral is work in progress so keep your eyes open for that too!

Re: Groqchat

#106

Post by tome »

mlazos wrote: post #63
GamerAlias wrote: post #27

In case, it's not blinding obvious to people. Groq are a hardware company that have built chips that are designed around the training and serving of machine models particularly targeted at LLMs. So the quality of the response isn't really what we're looking for here. We're looking for speed i.e. tokens per second. I actually have a final round interview with a subsidiary of Groq coming up and I'm very undecided as to…

tbh anyone can build fast hw for a single model, I’d audit their plan for a SW stack before joining. That said their arch is pretty unique so if they’re able to get these speeds it is pretty compelling

Our hardware architecture was not designed with LLMs in mind, let alone a specific model. It's a general purpose numerical compute fabric. Our compiler allows us to quickly deploy new models of any architecture without the need that graphics processors have for handwritten kernels. We run language models, speech models, image generation models, scientific numerical programs including for drug discovery, ...

Re: Groqchat

#107

Post by tome »

coder543 wrote: post #17

Is there any plan to show what this hardware can do for Mixtral-8x7B-Instruct? Based on the leaderboards[0], it is a better model than Llama2-70B, and I’m sure the T/s would be crazy high. [0]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Yup, deploying Mixtral is a work in progress. Watch this space!

Re: Groqchat

#108

Post by tome »

huevosabio wrote: post #42

I saw this in person back in September. Really impressed by their hardware. I'm still wondering why is the uptake so slow. My understanding from their presentations was that it was relatively simple to compile a model. Why isn't it more talked about? And why not demo Mixtral or show case multiple models?

We're building out racks as fast as we can to keep up with customer demand :) A public demo of Mixtral is in the works, so watch this space.

Re: Groqchat

#109

Post by tome »

chihuahua wrote: post #85
razorguymania wrote: post #25

Earlier quoted context omitted.

Its using vanilla llama-2 from Meta with no fine tuning. The point here is the speed and responsiveness of the underlying HW and SW.

But if the quality of the response is poor, it's irrelevant that it was generated quickly. If it was using different data to generate higher quality responses, would that not slow it down?

nomel gave a good answer in a different thread

> This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo.

To compare apples to apples look at the tokens per second of other systems running Llama 2 70B 4096. We're by far the fastest!

https://news.ycombinator.com/item?id=38742466

Re: Groqchat

#110

Post by tome »

kristjansson wrote: post #77

There was a good talk at HC34 about the accelerator Groq was working on at the time. I’m just a lay observer so I don’t know how much of that architecture maps to this new product, but it gives some insight into their thinking and design. https://youtu.be/MWQNjyEULDE?si=lBk6a_7DTNKOd8e7&t=62

Thanks for sharing. It's the same silicon architecture as in that talk. We have built out different system architectures based on that silicon, and this is our fastest one so far for LLMs. Expect to see even more speed increases soon!
Post reply on HN