Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

31–40 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#31
post #21

Jesus that makes chatgpt and even gemini seem slow AF

better quality than I was expecting. For fun I set the system prompt to:

You are a leader of a team of ai helpers. when given a question you can call on an expert, as a wizard calls on magic. You will say, I call forth {expert} master of {subject matter} an expert in {x, y, z}. Then you will switch to that persona.

I was not let down..

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#33

Incredible job. Feels dumb or obvious to say this, but this really changes the way I think of using it. The slow autoregression really sucks because it inhibits your ability to skim sections. For me, that creates an unnatural reading environment. This makes chatgpt feel antiqued.

Yes, agreed. We believe the benefits of reducing latency are non-linear. You can hit different phase changes as the latency reduces and new applications become viable. Roundtripping text-to-speech and speech-to-text is one example. We're looking forward to seeing what low latency applications are unlocked by our new users!

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#34

What's the underlying hardware for this?

It's a system built from hundreds of GroqChips (a custom ASIC we designed). We call it the LPU (language processing unit). Unlike graphics processors, which are still best in class for training, LPUs are best in class for low latency and high throughput inference. Our LLMs are running on several racks with fast interconnect between the chips.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#35
post #15
post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

Impressive speed. Are there any plans to run fine-tuned models?

Yes, we're working on a feature to give our partners the ability to deploy their own fine-tuned models.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#36
post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

You're running fp32 models, fp16 or quantized?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#37
post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

are your accelerator chips designed in-house? or they're some specialized silicon or FPGPU or something that you wrote very optimized code for inference?

it's really amazing! the first time I tried the demo, I had to try a few prompts to believe it wasn't just an animation :)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#38
Interesting, I ran my "prime test", it passed on generating a prime, it failed on it being large.

> "Can you generate me a large prime please"

> Sure, here's a large prime number: 193707721

> This number is a prime number because it is only divisible by 1 and itself. You can check its primality by trying to divide it by all the numbers up to its square root, and if none of them divide it evenly, then it is a prime number. This number is quite large, but there are much larger prime numbers known. For example, the largest known prime number, as of January 2022, is 2^82,589,933−1, a number with 24,862,048 digits, discovered as part of the Great Internet Mersenne Prime Search (GIMPS).

I then prompted "That's not large" which seemed to break it, the response is stuck in a loop:

> That isn't large

> I apologize if the prime number I provided did not meet your expectations for "large." Prime numbers can be very large indeed, and there is no upper limit to how large a prime number can be.

If you are looking for a very large prime number, here is one with a billion digits:

100000303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303...

It's printed many more pages of 303030... but hacker news has a post length limit!

Further testing, such as prompting to generate a 1024 bit prime also runs into odd loops.

> Can you generate a 1024 bit prime?

> Yes, I can generate a 1024-bit prime number for you. Here's one I just generated:

> 115322512531875133213135311402305250340282302283103413642283221253115653221253112210221122112112112112112112112112112112112112112112112112112112112112112112112112112112112112112112112112112112 ( Loops 112 forever )

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#40
post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

Alright, I'll bite. Haskell seems pretty unique in the ML space! Any unique benefits to this decision, and would you recommend it for others? What areas of your project do/don't use Haskell?
Post reply on HN