Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who uses Google TPUs for inference in production?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Who uses Google TPUs for inference in production?

#31

Post by ethbr1 »

htrp wrote: post #19
mike_d wrote: post #18

Earlier quoted context omitted.

> I think the tech stack needs another 12-18 months to mature Google has been doing AI before any other company even thought about it. They are on the 6th generation of TPU hardware. I don't think there is any maturity issue, just an availability issue because they are all being used internally.

100% agree, if I have access to the TPU team internally, it'll be very easy to use in production. If you aren't internal, the documentation, support, and even just general bug fixing is impossible to get.

(Has an expert team dedicated solely to optimizing for exotic hardware) = an option

(Doesn't have a team like that) = stick to mass-use, commodity hardware

That's generally been the trade-off since ~1970. And usually, the performance isn't worth the people-salaries.

How many examples of successful hardware that isn't well-documented and doesn't have drop-in 1:1 SDK coverage vs (more popular solution) are there?

It seems like a heavy-lift to even get something that does have parity in those ways adopted, given you're fighting market inertia.

Re: Who uses Google TPUs for inference in production?

#32

Post by rj45jackattack »

natbobc wrote: post #28
_b wrote: post #22

Earlier quoted context omitted.

I think this is right, in part because I've been told exactly this from people who work for Google and their job is to sell me cloud stuff- i.e., they say they have so much internal demand they aren't pushing TPUs for external use. Hence external pricing and support just isn't that great right now. But presumably when capacity catches up they'll start pushing TPUs again.

Feels like a bad point in the curve to try and sell them. “Oh our internal hypecycle is done… we’ll put them in the market now that they’re all worn out.

Sounds like ButterflyLabs.

Re: Who uses Google TPUs for inference in production?

#33

Post by chatmasta »

mike_d wrote: post #18
htrp wrote: post #6

We've previously tried and almost always regretted the decision. I think the tech stack needs another 12-18 months to mature (doesn't help that almost all work ex Google is being done in torch).

> I think the tech stack needs another 12-18 months to mature Google has been doing AI before any other company even thought about it. They are on the 6th generation of TPU hardware. I don't think there is any maturity issue, just an availability issue because they are all being used internally.

Google sells access to TPUs in its cloud platform, so you'd think they would be more open about sharing development and tooling frameworks for TPUs. It's like Borg (closed source, never used outside Google, made them no profit) vs. Kubernetes (open source, used everywhere, makes them profit).

Re: Who uses Google TPUs for inference in production?

#34

Post by Havoc »

pogue wrote: post #29

I've seen people connecting these to Raspberry Pis to run local LLMs but I'm not sure how effective it is. Check YouTube for some videos about it. Speaking of SBCs, prior to the Raspberry Pi, I was looking at the Orange Pi 5 which has a Rockchip RK3588S with an NPU (Neural Processing Unit). This was the first I had heard of such a thing but I was curious how/what exactly it does. Unfortunately, there's very little su…

The rock chip npu can do object recognition a la opencv but not LLMs

Re: Who uses Google TPUs for inference in production?

#35

Post by ein0p »

They aren’t really an alternative to anything. For one thing they’re now often slower on per-accelerator basis than NVIDIA stuff. They’re cheaper, of course, but because of disparity in performance you’ll need to estimate cost per flop on your own particular workload. They are also more difficult and slower to develop against, and SWE cost is always an issue if you don’t own a money printer like Google. Furthermore, for advanced users who can do their own CUDA kernels or Triton, that too can unlock additional efficiency from GPU. Such capability can’t even be contemplated on the TPU side because you basically get a black box. Then there’s the issue of limited capacity, further exacerbated by the fact that this capacity is provided by a single supplier who is struggling to fulfill its internal needs (which is why you can’t get v5). You can’t just get TPUs elsewhere. You can’t get them under your desk for dev work either.

That said, it wouldn’t be too difficult to port most models to Jax, load in the existing weights, and export the result for serving. Should you bother? IMO, no, unless we’re talking really large scale inference. Your time and money are almost certainly better spent iterating on the models.

Re: Who uses Google TPUs for inference in production?

#36

Post by mianos »

ooterness wrote: post #23

There's a cubesat using a Coral TPU for pose estimation. https://aerospace.org/article/aerospaces-slingshot-1-demonst...

They were lucky to get that going. The software support for the USB TPU was abandoned by google years ago now. Works fine if you run ubuntu 16 I think.

Re: Who uses Google TPUs for inference in production?

#37

Post by fnord77 »

mike_d wrote: post #18
htrp wrote: post #6

We've previously tried and almost always regretted the decision. I think the tech stack needs another 12-18 months to mature (doesn't help that almost all work ex Google is being done in torch).

> I think the tech stack needs another 12-18 months to mature Google has been doing AI before any other company even thought about it. They are on the 6th generation of TPU hardware. I don't think there is any maturity issue, just an availability issue because they are all being used internally.

> Google has been doing AI before any other company even thought about it

This not even remotely true. SRI was working on AI in various forms long before google existed

Re: Who uses Google TPUs for inference in production?

#38

Post by hcfman »

htrp wrote: post #20
emadm wrote: post #3

https://pytorch.org/blog/high-performance-llama-2/

>Cheers, > The PyTorch/XLA Team at Google Meanwhile you have an issue from 5 years ago with 0 support https://github.com/pytorch/xla/issues/202

5 years ago PyTorch wasn’t owned by the Linux foundation. Give ‘em a chance now.

On my wish list for PyTorch is that the apt install version work out of the box on Jetson SBCs

Re: Who uses Google TPUs for inference in production?

#39

Post by juliensalinas »

We tried hard to move some of our inference workloads to TPUs at NLP Cloud, but finally gave up (at least for the moment) basically for the reasons you mention. We now only perform our fine-tunings on TPUs using JAX (see https://nlpcloud.com/how-to-fine-tune-llama-openllama-xgen-w...) and we are happy like that.

It seems to me that Google does not really want to sell TPUs but only showcase their AI work and maybe get some early adopters feedback. It must be quite a challenge for them to create a dynamic community around JAX and TPUs if TPUs stay a vendor locked-in product...

Re: Who uses Google TPUs for inference in production?

#40

Post by arthurdelerue »

htrp wrote: post #6

We've previously tried and almost always regretted the decision. I think the tech stack needs another 12-18 months to mature (doesn't help that almost all work ex Google is being done in torch).

I agree with that, and I'm not sure they'll be able to improve the stack dramatically by themselves without the open-source community being more involved.
Post reply on HN