Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers on Chips

etched.ai

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Transformers on Chips

#11

Post by lumost »

andy99 wrote: post #7

There is a lot going on in the LLM / AI chip space. Most of the big players are focusing on general purpose AI chips, like Cerebras and Untether. This - what I understand to be more like ASICs is an interesting market. They give up flexibility but presumably can make them more cheaply. There is also Positron AI in this space, mentioned here: https://news.ycombinator.com/item?id=38601761 I'm only peripherally aware of…

In the cloud, these chips will compete head to head with GPUs. If they are able to pull off a 10x price/performance win without excessive porting work… it’ll take off in a heartbeat.

Re: Transformers on Chips

#12

Post by Me1000 »

masterofsome wrote: post #8

Isn't this kinda pigeonholing yourself to one neural network architecture? Are we sure that transformers will take us to the promised land? Chip design is a pretty expensive and time consuming process, so if a new architecture comes out that is sufficiently different from the current transformer model wouldn't they have to design a completely new chip? The compute unit design is probably similar from architecture to…

This was my first thought too. Even if transformers turn out to be the holy grail for LLMs, people are still interested in diffusion models for image generation.

I think we’re about to see a lot of interesting specialized silicon for neural nets in the coming years, but locking yourself into a specific kind of model seems a little too specialized right now.

Re: Transformers on Chips

#13

Post by Zenst »

andy99 wrote: post #7

There is a lot going on in the LLM / AI chip space. Most of the big players are focusing on general purpose AI chips, like Cerebras and Untether. This - what I understand to be more like ASICs is an interesting market. They give up flexibility but presumably can make them more cheaply. There is also Positron AI in this space, mentioned here: https://news.ycombinator.com/item?id=38601761 I'm only peripherally aware of…

Only one certainty, HBM memory makers will be doing nicely in the current climate as all these AI processing options are using it in larger and larger volumes. Those will be the unnoticed winners in this rush.

Re: Transformers on Chips

#14

Post by quickthrower2 »

masterofsome wrote: post #8

Isn't this kinda pigeonholing yourself to one neural network architecture? Are we sure that transformers will take us to the promised land? Chip design is a pretty expensive and time consuming process, so if a new architecture comes out that is sufficiently different from the current transformer model wouldn't they have to design a completely new chip? The compute unit design is probably similar from architecture to…

It’s a bet. Probably a good one to make. The upside of being the ones who have an AI chip (not a graphics chip larping as an AI chip) is huge. It will run faster and more cheaply. You get to step all over OpenAI, or get a multi billion dollar deal to supply Microsoft data centres. Or these ship on every new laptop etc. You get to be the next unicorn ($1tn company). So that is a decent bet for investors assuming the team can deliver. Yes the danger is there is a new architecture that runs on a CPU that for practical purposes whoop’s Attention’s ass. In which case investors can throw some money at ASICifying that.

Re: Transformers on Chips

#15

Post by Zenst »

lumost wrote: post #11
andy99 wrote: post #7

There is a lot going on in the LLM / AI chip space. Most of the big players are focusing on general purpose AI chips, like Cerebras and Untether. This - what I understand to be more like ASICs is an interesting market. They give up flexibility but presumably can make them more cheaply. There is also Positron AI in this space, mentioned here: https://news.ycombinator.com/item?id=38601761 I'm only peripherally aware of…

In the cloud, these chips will compete head to head with GPUs. If they are able to pull off a 10x price/performance win without excessive porting work… it’ll take off in a heartbeat.

Like ASIC Botcoin miners did. There are parallels here in how it might just pan out.

Re: Transformers on Chips

#16

Post by andy_xor_andrew »

fnbr wrote: post #10
andy_xor_andrew wrote: post #6

interesting how MCTS decoding is called out. that seems entirely like a software aspect, which doesn't depend on a particular chip design? and on the topic of MCTS decoding, I've heard lots of smart people suggest it, but I've yet to see any serious implementation of it. it seems like such an obviously good way to select tokens, you'd think it would be standard in vllm, TGI, llama.cpp, etc. But none of them seem to u…

It’s very difficult to implement, and requires training the network to use it. I worked at DeepMind on projects that used MCTS. Even with access to the AlphaZero source code, it was very difficult to write an other implementation that got the same results as the original.

I'm really curious about this part:

> and requires training the network to use it.

I thought one of the benefits of MCTS was, if you already have your value network, then a general MCTS implementation can walk the tree of values created by that network. And so no special update to the model is necessary. But I'm probably wrong about this.

(also, it boosts my confidence to hear that even folks at DeepMind find MCTS difficult to implement :D Because I tried to implement a simple MCTS a few years back for a very small toy project. I was following a step-by-step explanation of how it worked, and even still, it was super difficult, and very prone to subtle bugs)

Re: Transformers on Chips

#17

Post by GaggiX »

masterofsome wrote: post #8

Isn't this kinda pigeonholing yourself to one neural network architecture? Are we sure that transformers will take us to the promised land? Chip design is a pretty expensive and time consuming process, so if a new architecture comes out that is sufficiently different from the current transformer model wouldn't they have to design a completely new chip? The compute unit design is probably similar from architecture to…

Well, GPT-4 runs on a transformer architecture, and even if for unknown reasons GPT-4 is the upper limit of what you can achieve with transformer models, having hardware specialized to run the architecture extremely fast would always be very useful for many tasks (the tasks GPT-4 can already handle at least).

Re: Transformers on Chips

#18

Post by rvz »

This only tells me we are at peak AI hype, given that products like this have to dress up ASICs as 'Transformers on Chips' or 'Transformer Supercomputer'.

As always, no technical reports or in-depth benchmarks other than a unlabelled chart comparing against Nvidia H100s with little context and marketing jargon to the untrained eye.

It seems that this would tie you into a specific neural net implementation (i.e llama.cpp as a ASIC) and would have to require a hardware design change to support another.

Post reply on HN