111–120 of 131 posts
Can someone explain th hardware differences for training vs inference? I believe Nvidia is still the leader in training?
For someone who is totally clueless, I can see it's faster than chat gpt in responding to the same question. What are some relevant speed metrics? Output tokens per second? How about number of input tokens -- does that matter/how does that factor in.
is the TSP a RISC-V on FPGA ? the tweet mentions haskell, which sounds familiar - Bluespec or something. or is it a completely custom ASIC
Earlier quoted context omitted.
Its not using GPUs. Its using Groq's own HW - LPUs
How much do they cost to buy? Could I get something reasonable for $20k?
Incredibly fast. I wonder if they've released verification that it matches llama 70b on regular hardware?
Earlier quoted context omitted.
I assume it queues up requests if there are too many in flight to handle all of them at the same time, but it shows you the tokens per second (T/s) for every response, which is the number that matters (and presumably won't include the time spent in the queue).
You can be confused by why people are interested in the outputs, or you can think that the explanation is adequate. You can't do both at once. Here, in fact, the explanation is not adequate. Let's analyze: > Welcome Groq® Prompster! Are you ready to experience the world's fastest Large Language Model (LLM)? "The world's fastest LLM? They must have made an LLM, I guess" > We'd suggest asking about a piece of history,…
Earlier quoted context omitted.
I'd assume it's because that's not explained very well in the linked page.
There is quite literally a modal pop-up that explains it, which you must dismiss before you can begin interacting with the demo. Quoting the pop-up: "This alpha demo lets you experience ultra-low latency performance using the foundational LLM, Llama 2, 70B created by Meta AI - running on Groq." Towards the bottom of the page, it also says "Model: Llama 2 70B/4096". Below that, it says "This is a Llama2-based chatbot.…
You can be wonder why people are interested in the outputs, or you can think that the explanation is adequate. You can't do both at once.
Here, in fact, the explanation is not adequate. Let's analyze:
> Welcome Groq® Prompster! Are you ready to experience the world's fastest Large Language Model (LLM)?
"The world's fastest LLM? They must have made an LLM, I guess"
> We'd suggest asking about a piece of history, requesting a recipe for the holiday season, or copy and pasting in some text to be translated; "Make it French."
"Instructions, ok".
> This alpha demo lets you experience ultra-low latency performance using the foundational LLM, Llama 2, 70B created by Meta AI - running on Groq.
"So it's not their own LLM? It's just LLaMa? What's interesting about that? And what's Groq, the thing this is 'running on'? The website? I kind of guessed that, by virtue of being here."
> Like any AI demo, accuracy, correctness, or appropriateness cannot be guaranteed.
"Ok, sure".
Nowhere does it say "we've built new, very fast processing hardware for LLMs. Here's a demo a bog-standard LLM, LLaMa 70B, running on our hardware. Notice how fast it is".
is the TSP a RISC-V on FPGA ? the tweet mentions haskell, which sounds familiar - Bluespec or something. or is it a completely custom ASIC
It's a completely custom ASIC. Haskell was used in the hardware design, in a Bluespec-like way. Some parts of the compiler tool chain and infrastructure are also written in Haskell. We have loads of C++ and Python too, as you would imagine.