211–220 of 482 posts
If the page can't access certain fonts, it will fail to work, while it keeps retrying requests: https://fonts.gstatic.com/s/notosansarabic/[...] https://fonts.gstatic.com/s/notosanshebrew/[...] https://fonts.gstatic.com/s/notosanssc/[...] (I noticed this because my browser blocks these de facto trackers by default.)
Same problem when trying to use font replacements with a privacy plugin. This is a very weird dependency to have :-)
Earlier quoted context omitted.
Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.
Thanks for the quick reply! About hardware support, I was wondering if the LPU has a hardware instruction to compute the attention matrix similar to the MatrixMultiply/Convolve instruction in the TPU ISA. (Maybe a hardware instruction which fuses a softmax on the matmul epilogue?)
Earlier quoted context omitted.
When will we be able to buy Groq accelerator cards that would be affordable for hobbyists?
We are prioritising building out whole systems at the moment I don't think we'll have a consumer level offering in the near future.
I'm coding to NVidia right now. That builds them a moat. The instant I can get other hardware working, the less of a moat they will have. The more open it is, the more likely I am to adopt it.
Earlier quoted context omitted.
At least for the earlier Llama 70B demo, they claimed to be running unquantized. https://twitter.com/lifebypixels/status/1757619926360096852 Update: This comment says "some data is stored as FP8 at rest" and I don't know what that means. https://news.ycombinator.com/item?id=39432025
The weights are quantized to FP8 when they're stored in memory, but all the activations are computed at full FP16 precision.
Earlier quoted context omitted.
It's a general purpose compute engine for numerical computing and linear algebra, so it can accelerate any ML workloads. Previously we've accelerated models for stabilising fusion reactions and for COVID drug discovery * https://alcf.anl.gov/news/researchers-accelerate-fusion-rese... * https://wow.groq.com/groq-accelerates-covid-drug-discovery-3...
So, is this specific chip only for LLMs as the name LPU suggests Language Processing Unit, right?
Earlier quoted context omitted.
West Coast US. You would have been placed in our queuing system because with all the attention we are getting we are very busy right now!
Thanks! I did notice the queue count showing up occasionally but not every time. Maybe someone could repeat the test who has access without the queue so we can get an understanding of the potential latency once scaled and geo-distributed. What I'm really trying to understand is time to first token output actually faster than GPT 3.5 via API or just the rate of token output once it begins.
https://github.com/ray-project/llmperf-leaderboard?tab=readm...
This demo has more than 500 chips btw, it’s not exactly an apples to apples comparison with 1 GPU…
Definitely not, but even with a comparison to 500 GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)
Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"?
https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/rel/do...
This chart from NVIDIA implies their H100 runs llama v2 70B at >500 tok/s.
