This is extremely impressive - no login, extremely fast, and Mixtral quality is very good. It's already more useful than my (paid) GPT4 for many things due to the speed.
Groq runs Mixtral 8x7B-32k with 500 T/s
471–480 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#472Earlier quoted context omitted.
I actually wasn't so much talking from that perspective, as simply from the perspective of the design of individual pieces. There were rather clever things done in e.g. older multipliers or adders or similar which, I think , could apply to most modern parallel architectures, be that GPGPU, SP, MPE, FPGA, or whatever, in order to significantly increase density at a cost of slightly reduced serial performance. For mach…
Truth is that there is another startup called graph core that is doing exactly that, and also a really big chip
I assume no one will read this, but good places to look for super-clever ways to reduce transistor count while maintaining good performance:
- Early mainframes / room-sized computers (era of vacuum tubes and discrete transistors), especially at the upper-end , where there was enough budget to have modern pipelined and scalar architectures.
- Cray X-MP and successors
- DEC Alpha / StrongARM (referenced SA-110)
Bad places to look are all the microcode architectures. These optimized transistor count, often sacrificing massive amounts of performance in order to save on cost. Ditto for some of the minicomputers, where the goal was to make an "affordable" computer. Something like the PDP was super-clever in cost-cutting, which made sense at the time, does much less to maintain performance.
There's a ton of long-forgotten cleverness.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#473Earlier quoted context omitted.
https://www.livecoinwatch.com/price/GroqAI-GROQ I suppose that's someone else then? If that's true, then with this and Elon's Grok it's surprising the US Patent office hasn't taken your trademark away yet for not adequately defending it from infringement.
I don't know what that is. It's nothing to do with Groq Inc.
That seems to be this: https://groqerc.com/#, which in turn links to groq.com
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#474Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
When you start using Samsung 4 nm are you switching from SRAM to HDM? If yes, how's that going to affect all the metrics given that SRAM is so much faster? Someone said you'll eventually move to HDM because SRAM improvements is relatively stalled.
I read an article that indicated your Bill Of Materials compared to NVidia's is 10x to get 1/10 the latency, and 8x BOM for throughput if Nvidia optimizes for throughput? Does this seem accurate? That CAPEX is the primary drawback?
https://www.semianalysis.com/p/groq-inference-tokenomics-spe...
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#475Sorry if this is dumb but how is this different to Elons Grok? Was Groq chosen as a joke or homage ?
I think the downvotes are because people expected you to know Grok is a software LLM where Groq is a hardware scheme on which to run LLMs, or that Groq came out about 7 years before Grok so the "homage" is the reverse, i.e. Elon possibly paying to Groq. Groq was chosen as homage to Heinlein's "Stranger in a Strange Land" which invented the word "Grok" to mean "to understand deeply and intuitively" but also eating a dead loved one ("Resident Alien" on Netflix did that this year). Elon just used the English word directly like "Windows" and "Apple". Not using the word directly like "Groq" makes internet searches easier for everyone.
BTW Heinlein described prompt engineering of an LLM perfectly throughout the opening chapter of his * 1966 * book "The Moon is a Harsh Mistress". The "engineer" even admits he had no hard-core "engineering" training because capitalizing on the new technology didn't need it. The chapter could have been written today.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#476This demo has more than 500 chips btw, it’s not exactly an apples to apples comparison with 1 GPU…
Definitely not, but even with a comparison to 500 GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)
> GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)
You literally can, in fact that’s the entire reason to use multiple chips.
See eg the TPU group’s paper: https://arxiv.org/abs/2211.05102
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#477Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Alright, I'll bite. Haskell seems pretty unique in the ML space! Any unique benefits to this decision, and would you recommend it for others? What areas of your project do/don't use Haskell?
Haskell is a great language. However, when you want to build production grade software it’s the wrong language. Specially when it comes to a complicated piece of software like the compiler for a novel chip. I can tell you for a fact it has always being the wrong choice (specially in the case of Groq).
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#478[flagged]
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#479Nevermind, I stand corrected. Blown tf away after trying the demo MYSELF. It's instantaneous, the last time I used an LLM that fast was a proprietary model with a small dataset. Lighting fast but it wasn't smart enough. This is wild. But I don't understand why the demo was so bad and why the demo took so long to respond to his questions?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#480Earlier quoted context omitted.
Alright, I'll bite. Haskell seems pretty unique in the ML space! Any unique benefits to this decision, and would you recommend it for others? What areas of your project do/don't use Haskell?
Haskell is a great language. However, when you want to build production grade software it’s the wrong language. Specially when it comes to a complicated piece of software like the compiler for a novel chip. I can tell you for a fact it has always being the wrong choice (specially in the case of Groq).
Why? Is it just because it's too hard to read and write? Or is it lacking some features?