Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers on Chips

etched.ai

Post reply on HN
Print view View original

61–65 of 65 posts

Re: Transformers on Chips

#61

Post by smugma »

WhitneyLand wrote: post #23

I am not buying this at all. But I’m not a hardware guy so maybe someone can help with why this is not true: - Crypto hardware needed SHA256 which is basically tons of bitwise operations. That’s way simpler than the tons of matrix ops transformers need. - NVidia wasn’t focused on crypto acceleration as a core competency. There are focussed on this, and are already years down the path. - One of the biggest bottlenecks…

https://www.eetimes.com/harvard-dropouts-raise-5-million-for...

“Uberti cites bitcoin mining chips as an example of a successful specialized ASIC offering.“

The founder also references crypto, so your comparison is an apt rebuttal to an argument you didn’t know they were making.

Overall, the article gives a small bit of detail, which is infinitely more than gleaned from the website.

Re: Transformers on Chips

#62

Post by pavelstoev »

sp332 wrote: post #48
pavelstoev wrote: post #40

Earlier quoted context omitted.

Not exactly idle but only at around 30% utilization on average (measured on a ~900 GPU cluster over ~25 days)

If it's at 30% utilization then it's "mostly idle".

I agree. I am surprised that many folks, not you of course, think that is okay.

Re: Transformers on Chips

#63

Post by WhitneyLand »

guberti wrote: post #41

Founder here! We're still in stealth, but I'll be able to share details and performance figures soon. Our first product is a bet on transformers. If we're right, there's enormous upside - being transformer-specific lets you get an order of magnitude more compute than more flexible accelerators (GPUs, TPUs). We're hiring - if the EV makes sense for you, reach out at gavin @ etched.ai

Hey Gavin, I regret how much skepticism you faced here especially from me.

I believe honest feedback is important but that it should be given in the most productive way possible.

I always want to see fellow entrepreneurs succeed here, and will definitely keep an open mind as you release more details. Best of luck!

Re: Transformers on Chips

#64

Post by andy_xor_andrew »

fnbr wrote: post #58
andy_xor_andrew wrote: post #16

Earlier quoted context omitted.

I'm really curious about this part: > and requires training the network to use it. I thought one of the benefits of MCTS was, if you already have your value network, then a general MCTS implementation can walk the tree of values created by that network. And so no special update to the model is necessary. But I'm probably wrong about this. (also, it boosts my confidence to hear that even folks at DeepMind find MCTS di…

Ah, well you could use a standard value network, but it’d end really slow, so you probably want to train a smaller one and rely on the implicit ensembling that MCTS does to make it better. In my experience, PUCT does a lot better than UCT, so you want to also have a prior network. You don’t have to train a new network, but in my experience, it works much better. I haven’t spent a ton of time using off the shelf netwo…

really interesting! thanks for the info!

Re: Transformers on Chips

#65

Post by qeternity »

dramlord wrote: post #44
qeternity wrote: post #36

Yeah I call BS on this. This does nothing to address the main issues with autoregressive transformer models (memory bandwidth). GPU compute units are mostly sitting idle these days waiting for chip cache to receive data fr VRAM. This does nothing to solve that.

You can amortize memory loading with large continuous batching. I imagine more compute would help the problem for certain workloads like speculative decoding

Batching helps throughput and anyone running in production will be doing batching.

But it's not free, and still comes at a cost of per-stream latency.

Speculative decoding seems less effective in practice than in theory.

Post reply on HN