251–260 of 482 posts
Do they make money from LLM service or by selling hardware? Homepage is confusing without any reference to other products.
Both, we sell tokens as a service and we sell enterprise systems.
Earlier quoted context omitted.
Definitely, that's why we've opened our API to everyone.
I don't think that quite does it. What I'd want -- if you want me to support you -- is access to the chip, libraries, and API documentation. Best-case would be something I buy for (The mistake a lot of services make is to limit free tier to e.g. 30 day or 1 year, rather than hours/month; if I didn't get around to evaluating, switch employers, switch projects, etc. the free tier is gone). I did sign up for your API se…
Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
And here are some independent benchmarks https://artificialanalysis.ai/models/llama-2-chat-70b
Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…
https://www.hpcwire.com/2023/08/17/nvidia-h100-are-550000-gp...
Earlier quoted context omitted.
Can you explain if this affects quality relative to fp16? And is mixtral quantized?
We don't think so, but you be the judge! I believe we quantize both Mixtral and Llama 2 in this way.
Earlier quoted context omitted.
I honestly don't see the problem. "just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.
per-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
I would expect the model loading to take basically zero percent of the time in the above workflow
Earlier quoted context omitted.
We don't think so, but you be the judge! I believe we quantize both Mixtral and Llama 2 in this way.
Is your confidence rooted in quantified testing, or just vibes? I'm sure you're right, just curious. (My reasoning: running inference at full fp16 is borderline wasteful. You can use q7 with almost no loss.)
Earlier quoted context omitted.
Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…
> Show me a 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia I built one, should be live soon ;-)