121–130 of 131 posts
May as well wait for the whole response and render it. Or render paragraph at a time.
Don’t jiggle the UI while rendering.
Earlier quoted context omitted.
It's a completely custom ASIC. Haskell was used in the hardware design, in a Bluespec-like way. Some parts of the compiler tool chain and infrastructure are also written in Haskell. We have loads of C++ and Python too, as you would imagine.
very cool. thanks for sharing. i would not have guessed haskell for the compiler tool chain. Why did you choose that ? i mean haskell has a LONG history in chip design...but compilers are usually the forte of llvm/c++, etc. im guessing it must have been non trivial to do this.
Earlier quoted context omitted.
How many chips are used for this demo? Do they have dram? I remember the earlier versions did not have dram. Are they also used for training or just inference?
I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total. Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!
Earlier quoted context omitted.
I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total. Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!
Could you explain the blockers to getting back-propagation working well on your chips?
Does that answer your question?
Earlier quoted context omitted.
Could you explain the blockers to getting back-propagation working well on your chips?
Our language processors have much lower latency and higher throughput than graphics processors so we have a massive advantage when it comes to inference. For language models particularly, time to first token is hugely important (and will probably become even more important as people start combining models to do novel things). Additionally, you probably care mostly about batch size 1. For training, latency is not the…
Training is both memory throughput and compute constrained. Much research in speeding up training goes into optimizing HBM to SRAM communication. The equivalent for your chips would be communication from the SRAM of one chip to the SRAM of another, where it sounds like your architecture has a major memory throughput advantage over GPUs. So I assume you don't have a proportional compute advantage?
By the way, it's great to see a non von Neumann architecture showing a major performance advantage in a real world application. And your chips are conceptually equivalent to chiplets; you should have a major cost advantage on bleeding edge process nodes if you scale up manufacturing. Overall very impressive!
Earlier quoted context omitted.
This is really impressive. For reference, inference for llama 70b on together’s api generates text at roughly 60 tokens/second. I can’t find any information about an api, though I’m guessing that the costs are eye watering. If they offered a Mixtral endpoint that did 300-400 tokens per second at a reasonable cost, I can’t imagine ever using another provider.
We don't have an API in public availability yet but that's coming soon in the new year. We will be price competitive with OpenAI but much faster. Deploying Mixtral is work in progress so keep your eyes open for that too!
Earlier quoted context omitted.
We don't have an API in public availability yet but that's coming soon in the new year. We will be price competitive with OpenAI but much faster. Deploying Mixtral is work in progress so keep your eyes open for that too!
Also make a long context Mistral-7B that spits 1000T/s
Earlier quoted context omitted.
Our language processors have much lower latency and higher throughput than graphics processors so we have a massive advantage when it comes to inference. For language models particularly, time to first token is hugely important (and will probably become even more important as people start combining models to do novel things). Additionally, you probably care mostly about batch size 1. For training, latency is not the…
Everything you say makes sense. Training is definitely more compute intensive than inference. Training is both memory throughput and compute constrained. Much research in speeding up training goes into optimizing HBM to SRAM communication. The equivalent for your chips would be communication from the SRAM of one chip to the SRAM of another, where it sounds like your architecture has a major memory throughput advantag…