No prompt length? For practical purposes, the prompt processing time would far more important.
We're going to add a selector to choose prompt size (and multimedia content in the prompt)
Thefastest.ai
21–30 of 36 posts
Re: Thefastest.ai
#22Re: Thefastest.ai
#23There are dozens of AI chip startups out there with wild claims about speed. Groq seems like the first to actually prove it by launching a product. I hope they spur a speed war with other chipmakers to make the fastest inference engine.
Re: Thefastest.ai
#24I'd be interested to hear how Llama 8B with long chain-of-thought prompts compares to GPT-4 one-shot prompts for real-world tasks.
In classification for example, you could ask Llama 8B to reason through each possibility, rank them, rate them, make counterarguments, etc. - all in the same time that GPT-4 would take to output one classification without reasoning. Which does better?
Re: Thefastest.ai
#25I'd be interested to hear how Llama 8B with long chain-of-thought prompts compares to GPT-4 one-shot prompts for real-world tasks. In classification for example, you could ask Llama 8B to reason through each possibility, rank them, rate them, make counterarguments, etc. - all in the same time that GPT-4 would take to output one classification without reasoning. Which does better?
Good idea, that could make for a pretty interesting eval. It's similar to a timed test... we don't really care how long it takes or how much scratch paper you needed as long as you deliver the correct answer within the time limit.
Re: Thefastest.ai
#26Groq really has an unfortunate name. (I assume they had theirs before Grok.)
The spelling with a 'k' is more canon (referring to the term from Heinlein) and that was the spelling in the tech culture that borrowed it... what is the reason for choosing a 'q' in theirs, do you know?
I like the "q" as in "query" or "question"; seems a fitting homophone.
Re: Thefastest.ai
#27Groq with llama3 70b is so fast and good enough for what we do (source code stuff) that it’s really quite painful to work with most others now. We replaced most our internal integrations with this and everything is great so far. I guess they will be bought soon?
What do you guys do?
Re: Thefastest.ai
#28I don't understanding why would we need to having similar expectations from systems that we have from humans and building a whole theory on it. I can adjust my behaviour around systems. I am not restricted to operate within default values. e.g Whenever a price is listed as $99, I automatically know it is $100. Marketing gimmicks don't work once you know about them or in other words, expectations can be set in a new e…
Marketing gimmicks absolutely still work even if you know about them because they take advantage of basic human psychology so when you're tired/hungry/sleepy or otherwise not operating at peak performance, your lizard brain/autopilot takes over and you choose what's been chosen for you.
Re: Thefastest.ai
#29I wish it also included latency of speech to text and text to speech APIs :)
Re: Thefastest.ai
#30Groq with llama3 70b is so fast and good enough for what we do (source code stuff) that it’s really quite painful to work with most others now. We replaced most our internal integrations with this and everything is great so far. I guess they will be bought soon?
How is that possible with Groq rate limits?
Fireworks has Llama 3 for the same effective speed with much more realistic rate limits (and billing)