Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670ms. Just wondering if that's an expected result, as it's on a par or slower than GPT 3.5 via API.
Groq runs Mixtral 8x7B-32k with 500 T/s
181–190 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#182I have this hooked up experimentally to my universal Dungeon Master simulator DungeonGod and it seems to work quite well.
I had been using Together AI Mixtral (which is serving the Hermes Mixtrals) and it is pretty snappy, but nothing close to Groq. I think the next closes that I've tested is Perplexity Labs Mixtral.
A key blocker in just hanging out a shingle for an open source AI project is the fear that anything that might scale will bankrupt you (or just be offline if you get any significant traction). I think we're nearing the phase that we could potentially just turn these things "on" and eat the reasonable inference fees to see what people engage with - with a pretty decently cool free tier available.
I'd add that the simulator does multiple calls to the api for one response to do analysis and function selection in the underlying python game engine, which Groq makes less of a problem as it's close to instant. This adds a pretty significant pause in the OpenAI version. Also since this simulator runs on Discord with multiple users, I've had problems in the past with 'user response storms' where the AI couldn't keep up. Also less of a problem with Groq.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#183I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…
I have it on good authority Apple was very closing to acquiring Groq
If this is true, expect a call from the SEC...
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#184Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
How do I sign up for API access? What payment methods do you support?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#185Earlier quoted context omitted.
Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior…
I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.
What do you mean by non-deterministic hardware? cuBLAS on a laptop GPU was deterministic when I tried it last iirc
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#186The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these perfor…
Thomas, I think for full disclosure you should also state that you left Groq to start a competitor (a competitor which doesn't have the world's lowest latency LLM engine nor a guarantee to match the cheapest per token prices, like Groq does.).
Anyone with a serious interest in the total cost of ownership of Groq's system is welcome to email contact@groq.com.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#187It's impressive, but I have one problem with all of those models. I wanted them to answer what Mixtral or Llama2 are, but with no luck. It would be great if models could at least describe themselves.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#188Earlier quoted context omitted.
I have it on good authority Apple was very closing to acquiring Groq
If this is true, expect a call from the SEC...
Even if it isn't true.
Disclosing inside information is illegal, _even if it is false and fabricated_, if it leads to personal gains.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#189Earlier quoted context omitted.
I can't think of any value in asking LLM's to generate prime numbers. They are neural nets, not turing machines.
The value is showing how confidently is presents incorrect results. Especially the lack of nuance or uncertainty in the language they use. They extremely confidently present the incorrect information, and prime generation is interesting because it's information that isn't easy to spot as obviously incorrect to the user while being information that's possible to determine is wrong at small numbers and difficult to ver…
Mixtral is a pre-existing model that was released by Mistral AI in December: https://mistral.ai/news/mixtral-of-experts/
It was discussed on HN at the time: https://news.ycombinator.com/item?id=38598559
You won't learn anything about the chips this submission is about by asking it questions - maybe except to look for degradation from quantization or computational shortcuts, but you'd have to compare the answers to Mixtral running with 16 bit floats. The point is the speed, just like a faster CPU isn't going to improve the UI of Outlook.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#190Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670ms. Just wondering if that's an expected result, as it's on a par or slower than GPT 3.5 via API.
There’s a queueing system if too many requests are being processed at once. You may have hit that.