Earlier quoted context omitted.
The speed part. We're not interested in being swallowed. The aim is to be bigger than Nvidia in three years :)
Can you warn us pre-IPO?
I'm sure you'll hear all about our IPO on HN :) :)
Groq runs Mixtral 8x7B-32k with 500 T/s
291–300 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#292Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
Maybe I'm stretching the analogy too far, but are we in the transistor regime of LLMs already? Sometimes I see these 70 billion parameter monstrosities and think we're still building ENIAC out of vacuum tubes. In other words, are we ready to steadily march on, improving LLM tok/s year by year, or are we a major breakthrough or two away before that can even happen?
[deleted]
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#293Earlier quoted context omitted.
I don't think that quite does it. What I'd want -- if you want me to support you -- is access to the chip, libraries, and API documentation. Best-case would be something I buy for (The mistake a lot of services make is to limit free tier to e.g. 30 day or 1 year, rather than hours/month; if I didn't get around to evaluating, switch employers, switch projects, etc. the free tier is gone). I did sign up for your API se…
Yup, understood. Access to consumer hardware like this is not something that we provide at the moment, I'm afraid.
Don't blame you. Been at plenty of startups, resources are finite, and focus is important.
My only point was to, well, perhaps bump this up from #100 on your personal priority list perhaps to #87, to the limited extent that influences your business.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#294Earlier quoted context omitted.
If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantee…
Surely once you're scaling over multiple chips/servers/racks you're dealing with retries and checksums and sequence numbers anyway? How do you get around the non-determinacy of networking beyond just hoping that you don't see any errors?
Our interconnect between chips is also deterministic! You can read more about our interconnect, synchronisation, and error correction in our paper.
https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#295Earlier quoted context omitted.
You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…
It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."
You're an idiot.
https://www.kiplinger.com/article/investing/t052-c008-s001-w...
Case #1.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#296Earlier quoted context omitted.
As tome mentioned we don’t quantize, all activations are FP16 And here are some independent benchmarks https://artificialanalysis.ai/models/llama-2-chat-70b
Jesus Christ, these speeds with FP16? That is simply insane.
Ask how much hardware is behind it.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#297Earlier quoted context omitted.
It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."
Had insider trading training, and yes, that's the gist of it. If you know or presume that the information is material (makes a difference) and not public, it's illegal to act on it.
Roughly, it's illegal only if you have some duty not to trade on it. If you acquired the information without misappropriating it (like overhearing it from strangers in a normal public bar), then you're free to trade.
https://corpgov.law.harvard.edu/2017/01/18/insider-trading-l...
There's no reason for normal corporate training to discuss that element, because an employee who trades their employer's stock based on MNPI has near-certainly misappropriated it. The question of whether a non-employee has misappropriated information is much more complex, though.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#298Earlier quoted context omitted.
It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."
Had insider trading training, and yes, that's the gist of it. If you know or presume that the information is material (makes a difference) and not public, it's illegal to act on it.
Training is designed to protect the corporation, not to provide accurate lega ladvice. That's true of most corporate trainings, for that matter, be that bribes / corruption, harassment, discrimination, or whatnot. Corporations want employees very far from the line.
That's the right way to run them.
If you want more nuance, talk to a lawyer or read case law.
Generally, insider trading requires something along the lines of a fiduciary duty to keep the information secret, albeit a very weak one. I'm not going to slice that line, but you see references in-thread.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#299Earlier quoted context omitted.
I don't know about HBM specifically, but DDR and GDDR at a protocol level are both deterministic. It's the memory controller doing a bunch of reordering that makes them non-deterministic. Presumably, if that is the reason you don't like DRAM, you could build your compiler to be memory-layout aware and have the memory controller issue commands without reordering.
Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
You can control when those happen, too.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#300This is super impressive. The rate of iteration and innovation in this space means that just as I'm feeling jaded/bored/oversaturated - some new project makes my jaw drop again.