21–30 of 65 posts
What about Transformers on FPGAs?
They haven't been able to compete with GPU's on perf/watt. In general you end up just designing some AI accelerator for the FPGA (because the models are too big to map onto a single device all at once), but it's hard to beat purpose-built tensor and vector HW on a GPU when you're running soft logic.
Earlier quoted context omitted.
In the cloud, these chips will compete head to head with GPUs. If they are able to pull off a 10x price/performance win without excessive porting work… it’ll take off in a heartbeat.
Like ASIC Botcoin miners did. There are parallels here in how it might just pan out.
- Crypto hardware needed SHA256 which is basically tons of bitwise operations. That’s way simpler than the tons of matrix ops transformers need.
- NVidia wasn’t focused on crypto acceleration as a core competency. There are focussed on this, and are already years down the path.
- One of the biggest bottlenecks is memory bandwidth. That is also not cheap or simple to do.
- Say they do have a great design. What process are they going to build it on? There are some big customers out there waiting for TMSC space already.
Maybe they have IP and it’s more of a patent play.
(I mention crypto only as an example of custom hardware competing with a GPU)
Such a chip (with support for LoRA finetuning) would likely be the enabler for the next-gen robotics.
Right now, there is a growing corpus of papers and demos that show what's possible, but these demos are often a talk-to-a-datacenter ordeal, which is not suitable for any serious production use: too high latency, too much dependency on the Internet.
With a low-latency, cost- and energy-efficient way to run finetuned LLMs locally (and keep finetuning based on the specific robot experience), we can actually make something useful in the real world.
I am not buying this at all. But I’m not a hardware guy so maybe someone can help with why this is not true: - Crypto hardware needed SHA256 which is basically tons of bitwise operations. That’s way simpler than the tons of matrix ops transformers need. - NVidia wasn’t focused on crypto acceleration as a core competency. There are focussed on this, and are already years down the path. - One of the biggest bottlenecks…
Isn't this kinda pigeonholing yourself to one neural network architecture? Are we sure that transformers will take us to the promised land? Chip design is a pretty expensive and time consuming process, so if a new architecture comes out that is sufficiently different from the current transformer model wouldn't they have to design a completely new chip? The compute unit design is probably similar from architecture to…
This was my first thought too. Even if transformers turn out to be the holy grail for LLMs, people are still interested in diffusion models for image generation. I think we’re about to see a lot of interesting specialized silicon for neural nets in the coming years, but locking yourself into a specific kind of model seems a little too specialized right now.
Isn't this kinda pigeonholing yourself to one neural network architecture? Are we sure that transformers will take us to the promised land? Chip design is a pretty expensive and time consuming process, so if a new architecture comes out that is sufficiently different from the current transformer model wouldn't they have to design a completely new chip? The compute unit design is probably similar from architecture to…
It’s a bet. Probably a good one to make. The upside of being the ones who have an AI chip (not a graphics chip larping as an AI chip) is huge. It will run faster and more cheaply. You get to step all over OpenAI, or get a multi billion dollar deal to supply Microsoft data centres. Or these ship on every new laptop etc. You get to be the next unicorn ($1tn company). So that is a decent bet for investors assuming the t…
Product page like this... they haven't even designed the chip. Complete vaporware.
Title was a bit of a letdown. I was hoping for a discussion of silicon planar transformers (like, the electrical component), which are of increasing interest in RF ICs. :)