Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers on Chips

etched.ai

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Transformers on Chips

#31

Post by treesciencebot »

WhitneyLand wrote: post #23

I am not buying this at all. But I’m not a hardware guy so maybe someone can help with why this is not true: - Crypto hardware needed SHA256 which is basically tons of bitwise operations. That’s way simpler than the tons of matrix ops transformers need. - NVidia wasn’t focused on crypto acceleration as a core competency. There are focussed on this, and are already years down the path. - One of the biggest bottlenecks…

> One of the biggest bottlenecks is memory bandwidth. That is also not cheap or simple to do.

This is precisely why people are trying to put logic into memory instead of just making the logic chips simpler. Compute being 10x faster doesn't mean much when you want real-time, near-zero latency in the current day (and potentially, future) ML workloads. Memory bandwith for low batches are much more important, and even though this chip comes with HBM3E (which is cutting edge), that by itself won't make this faster than H200/MI300X.

Re: Transformers on Chips

#32

Post by OJFord »

Where did this come from? There is absolutely nothing clickable except 'contact us' which just reloads the same page? There's almost zero information here?

Re: Transformers on Chips

#34

Post by mikewarot »

29athrowaway wrote: post #19

What about Transformers on FPGAs?

FPGAs are designed to fight latency as much as possible. To do this, they have networks of switches to shuttle bits across the chip and keep delays to the bare minimum, in order for synchronous logic to be able to run at the highest possible clock rates for signals that traverse the entire chip.

To meet this goal, there's a huge amount of effort required to compile a program written in Verilog, VHDL, etc.. into a set of bits that can be used to program all of the switching logic and look up tables in the chip. I'm lead to believe it can sometimes take a day or more per compile.

The second factor optimized for in FPGAs is utilization, trying to use 100% of the available resources of the chip. This is never achieved in practice.

Because everything is optimized for speed, it's not very power efficient.

---

Generally, FPGAs aren't the right architecture for neural networks. If you could load all of the weights into the LUTs, and leave them there, you'd get the type of speedups you want, but those scales of FPGA just don't exist.

Re: Transformers on Chips

#35

Post by [deleted] »

dontwearitout wrote: post #27
quickthrower2 wrote: post #14

Earlier quoted context omitted.

It’s a bet. Probably a good one to make. The upside of being the ones who have an AI chip (not a graphics chip larping as an AI chip) is huge. It will run faster and more cheaply. You get to step all over OpenAI, or get a multi billion dollar deal to supply Microsoft data centres. Or these ship on every new laptop etc. You get to be the next unicorn ($1tn company). So that is a decent bet for investors assuming the t…

Yep, transformers showed up in 2017, nearly 7 years ago, and they still wear the crown. Maybe some new architecture will come to dominate eventually, but I would love a low cost PCIe board that could run 80B transformer models today.

[deleted]

Re: Transformers on Chips

#36

Post by qeternity »

Yeah I call BS on this. This does nothing to address the main issues with autoregressive transformer models (memory bandwidth).

GPU compute units are mostly sitting idle these days waiting for chip cache to receive data fr VRAM.

This does nothing to solve that.

Re: Transformers on Chips

#37

Post by lumost »

chabons wrote: post #22
Zenst wrote: post #15

Earlier quoted context omitted.

Like ASIC Botcoin miners did. There are parallels here in how it might just pan out.

Interesting point. That said, the AI model space is rapidly evolving, while bitcoin's hashing problem is static. This makes it significantly more risky to make a large capital investment in dedicated HW when it's unclear if it will be able to run the next big model architecture. For instance, if this had been built + released a year ago, before SOTA models used MoE , then it would rapidly have become obselete.

Outside of hardware/implementation optimizations, and position embedding choice - has the SOTA transformer architecture evolved that much?

Llama-2 code appears to be about the same as gpt-2.

Re: Transformers on Chips

#38

Post by teaearlgraycold »

Could probably go even faster burning GPT-4's weights right into the silicon. No need to even load weights into memory.

Granted, that eliminates the ability to update the model. But if you already have a model you like that's not a problem.

Re: Transformers on Chips

#39

Post by Osiris »

OJFord wrote: post #32

Where did this come from? There is absolutely nothing clickable except 'contact us' which just reloads the same page? There's almost zero information here?

Maude you have JS disabled? It’s one of those fancy animations as you scroll websites.

Re: Transformers on Chips

#40

Post by pavelstoev »

qeternity wrote: post #36

Yeah I call BS on this. This does nothing to address the main issues with autoregressive transformer models (memory bandwidth). GPU compute units are mostly sitting idle these days waiting for chip cache to receive data fr VRAM. This does nothing to solve that.

Not exactly idle but only at around 30% utilization on average (measured on a ~900 GPU cluster over ~25 days)
Post reply on HN