I asked it "What is the oldest known written recipe" and then asked for that recipe. It seemed to repeat the same response a bunch of times (very quickly!) before stopping. Is this abnormal behavior?
Groq runs Mixtral 8x7B-32k with 500 T/s
151–160 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#152Earlier quoted context omitted.
It was discussed here on HN, that's how I know about it. I found a few things when searched around but not sure which one was the one I recall. Anyway, here is a video from one: https://www.youtube.com/watch?v=C2rbym6bXM0 Here is a paper discussing something similar: https://arxiv.org/abs/2307.07162
The description for that video says > Ghost Autonomy’s MLLM-based capabilities are currently in development. These video and image examples show MLLM-based analysis of driving scenes captured from Ghost vehicles driving in both autonomous and conventional mode. MLLM-based reasoning is not yet being returned to the car to impact actual driving maneuvers. So the model discussed is not doing any driving whatsoever. This…
Then its not the one I remember maybe.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#153In how far is the API compatible with OpenAI? Does it offer logprobs[0] and top_logprobs[1]? 0. https://platform.openai.com/docs/api-reference/chat/create#c... 1. https://platform.openai.com/docs/api-reference/chat/create#c...
You can find our API docs here, including details of our OpenAI compatibility https://docs.api.groq.com/
Filled the form for API Access last night. Is there a delay with increased demand now?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#154Earlier quoted context omitted.
No, it is "do whatever you were already doing with ML, faster" This question seems either from a place of deep confusion or is in bad faith. This post is about hardware. The hardware is model independent.* Any issues with models, like hallucinations, are going to be identical if it is run on this platform or a bunch of Nvidia GPUs. Performance in terms of hardware speed and efficiency are orthogonal to performance in…
Well ok, Groq provides lower latency cheaper access to the same models of questionable quality. Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.
Mixtral 8x7b is competitive with ChatGPT 3.5 Turbo so I'm not sure why you are being so dismissive.
https://chat.lmsys.org/ check the leaderboard.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#155Nice… a startup that has two “C” positions CEO and Chief Legal Officer… That sounds like a fun place to be
They seem to be around since 2016. May be not bad for a LLM company that would need to deal with legal issues?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#156Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#157Earlier quoted context omitted.
Yeah just offload the compute onto the cloud.
Its too unreliable, too restricted and too not-private. ChatGPT stopped processing images for me, trying to get help but support doesn't appear to be very fast, they asked for more info but not heard back since. Its too restricted, can't do anything on hard topics. It doesn't work when you try to work out exploits or dangers in a system for example. Its not private, they say they don't train on API requests but compa…
The model being too restrictive does seem to be a good point.
Do you think there are less restrictive models hosted on poe.com?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#158Earlier quoted context omitted.
We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.
Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior…
I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#159Earlier quoted context omitted.
If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantee…
I don't know about HBM specifically, but DDR and GDDR at a protocol level are both deterministic. It's the memory controller doing a bunch of reordering that makes them non-deterministic. Presumably, if that is the reason you don't like DRAM, you could build your compiler to be memory-layout aware and have the memory controller issue commands without reordering.
Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#160Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
How well would your hardware work for image/video generation?