I see XTX is one of the investors. Any potential other use cases with async logic beyond just inference?
See this reply: https://news.ycombinator.com/item?id=39437239
Groq runs Mixtral 8x7B-32k with 500 T/s
391–400 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#392I’m sure Elon is pissed since he has Grok. Someone now needs to make a Groc
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#393Earlier quoted context omitted.
> 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia. I believe that this is doable - my pipeline is generally closer to 400ms without RAG and with Mixtral, with a lot of non-ML hacks to get there. It would also definitely be doable with a joint speech-language model that removes the transcription step. For these use cases, time to first byte is the…
It’s important…if you’re building a chatbot. The most interesting applications of LLMs are not chatbots.
> The most interesting applications of LLMs are not chatbots.
What are they then? Every use case I’ve seen is either a chatbot or like a copy editor which is just a long form chatbot.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#394This is unreal. I have never seen anything this fast. How ? I mean, how can you physically ship the bits this fast, let alone a LLM.
Something about the UI. Doesn't work for me. May be I like openAI chat interface too much. Can someone bring their own data and train ? That would be crazy!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#395The demo is pretty cool, but the mobile interface could be a parody of bad interface design. The text box at the top is hard to reach if you want to open the keyboard, which automatically closes, or press the button to send the question, and the chat history is out of chronological order for no logical reason. Edit: Text selection is also broken.
> Edit: Text selection is also broken. Or disabled?
It works for me, but the selected text is superimposed on top of the normal text in a different size.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#396Impressive demo! However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? Also, while the CNN Vapi demo was impressive as well, a few weeks ago here[1] someone shared https://smarterchild.chat/ . That also has _very_ low audio latency, making natural conversation possible. From that discussion it seems tha…
> However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? For API access to our tokens as a service we guarantee to beat any other provider on cost per token (see https://wow.groq.com ). In terms of selling hardware, we're focused on selling whole systems, and they're only really suitable for corporations…
How open is your early access? i.e. likelihood to get API access granted right now
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#397Earlier quoted context omitted.
> 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia. I believe that this is doable - my pipeline is generally closer to 400ms without RAG and with Mixtral, with a lot of non-ML hacks to get there. It would also definitely be doable with a joint speech-language model that removes the transcription step. For these use cases, time to first byte is the…
It’s important…if you’re building a chatbot. The most interesting applications of LLMs are not chatbots.
> The most interesting applications of LLMs are not chatbots.
In your opinion, what are the most interesting?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#398Earlier quoted context omitted.
Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…
I have one, with 13B, on a 5-year-old 48GB Q8000 GPU. It’s also can see, it’s LLaVA. And it is very important that it is local, as privacy is important and streaming images to the cloud is time consuming. You only need a few tokens, not the full 500 tokens response to run TTS. And you can pre-generate responses online, as ASR is still in progress. With a bit of clever engineering the response starts with virtually no…
Did you find anything cheaper for local installation?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#399Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#400As a virtual reality geek, this is super exciting because although there are numerous people experimenting with voicing NPCs with LLMs, they all have horrible latency and are unusable in practice. This looks like the first one that can actually potentially work for an application like that. I can see it won't be long before we can have open ended realistic conversations with "real" simulated people!