Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who uses Google TPUs for inference in production?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Who uses Google TPUs for inference in production?

#41

Post by arthurdelerue »

ein0p wrote: post #35

They aren’t really an alternative to anything. For one thing they’re now often slower on per-accelerator basis than NVIDIA stuff. They’re cheaper, of course, but because of disparity in performance you’ll need to estimate cost per flop on your own particular workload. They are also more difficult and slower to develop against, and SWE cost is always an issue if you don’t own a money printer like Google. Furthermore,…

I agree, except about this statement: "it wouldn’t be too difficult to port most models to Jax"

--> We tried such ports at https://kwatch.io (the company I work for), and it appeared to be much harder than expected (at least for us). I don't think so many people are capable of porting an LLM based on PyTorch + GPU to Jax + TPU.

Re: Who uses Google TPUs for inference in production?

#42

Post by ein0p »

arthurdelerue wrote: post #41
ein0p wrote: post #35

They aren’t really an alternative to anything. For one thing they’re now often slower on per-accelerator basis than NVIDIA stuff. They’re cheaper, of course, but because of disparity in performance you’ll need to estimate cost per flop on your own particular workload. They are also more difficult and slower to develop against, and SWE cost is always an issue if you don’t own a money printer like Google. Furthermore,…

I agree, except about this statement: "it wouldn’t be too difficult to port most models to Jax" --> We tried such ports at https://kwatch.io (the company I work for), and it appeared to be much harder than expected (at least for us). I don't think so many people are capable of porting an LLM based on PyTorch + GPU to Jax + TPU.

Well, I should have said “it wouldn’t be too difficult for me” then. I keep forgetting why I get paid so much.

Re: Who uses Google TPUs for inference in production?

#43

Post by VirusNewbie »

ein0p wrote: post #42
arthurdelerue wrote: post #41

Earlier quoted context omitted.

I agree, except about this statement: "it wouldn’t be too difficult to port most models to Jax" --> We tried such ports at https://kwatch.io (the company I work for), and it appeared to be much harder than expected (at least for us). I don't think so many people are capable of porting an LLM based on PyTorch + GPU to Jax + TPU.

Well, I should have said “it wouldn’t be too difficult for me” then. I keep forgetting why I get paid so much.

I would love for you to expound, I found it interesting that you qualified your "should you bother, no" with "unless you are doing inference at scale". But in the previous paragraph you explained why you can get better performance with GPUs.

So is there some advantage of TPU, assuming there was SWE/API parity between GPUs?

Re: Who uses Google TPUs for inference in production?

#44

Post by ein0p »

VirusNewbie wrote: post #43
ein0p wrote: post #42

Earlier quoted context omitted.

Well, I should have said “it wouldn’t be too difficult for me” then. I keep forgetting why I get paid so much.

I would love for you to expound, I found it interesting that you qualified your "should you bother, no" with "unless you are doing inference at scale". But in the previous paragraph you explained why you can get better performance with GPUs. So is there some advantage of TPU, assuming there was SWE/API parity between GPUs?

Could be cheaper, depending on workload, and if you’re large that could justify the cost of additional SWE time required to port and support. Triton/CUDA requires people who know both DL and low level programming. Whether you get better performance _per dollar_ really depends on workload and also on the size of your workload. Here I don’t just mean the cost of buying compute in cloud, I mean the more broad definition: total cost of doing business, all in, including SWE cost. If you’re huge (eg Anthropic), SWE cost at scale is a lot easier to justify. If you’re on the smaller side, SWE cost matters a lot more. It’s way easier to hire PyTorch people (market share 60%) than eg Jax (market share 3%). And yeah I know there’s Torch XLA, but it’s basically the same thing with a different frontend.

Re: Who uses Google TPUs for inference in production?

#45

Post by TwentyPosts »

fnord77 wrote: post #37
mike_d wrote: post #18

Earlier quoted context omitted.

> I think the tech stack needs another 12-18 months to mature Google has been doing AI before any other company even thought about it. They are on the 6th generation of TPU hardware. I don't think there is any maturity issue, just an availability issue because they are all being used internally.

> Google has been doing AI before any other company even thought about it This not even remotely true. SRI was working on AI in various forms long before google existed

Who or what is SRI?

Re: Who uses Google TPUs for inference in production?

#46

Post by emu »

ein0p wrote: post #35

They aren’t really an alternative to anything. For one thing they’re now often slower on per-accelerator basis than NVIDIA stuff. They’re cheaper, of course, but because of disparity in performance you’ll need to estimate cost per flop on your own particular workload. They are also more difficult and slower to develop against, and SWE cost is always an issue if you don’t own a money printer like Google. Furthermore,…

> Such capability can’t even be contemplated on the TPU side because you basically get a black box.

I'll just leave this here: https://jax.readthedocs.io/en/latest/pallas/index.html

Re: Who uses Google TPUs for inference in production?

#47

Post by fluidcruft »

TwentyPosts wrote: post #45
fnord77 wrote: post #37

Earlier quoted context omitted.

> Google has been doing AI before any other company even thought about it This not even remotely true. SRI was working on AI in various forms long before google existed

Who or what is SRI?

See http://www.sri.com

Re: Who uses Google TPUs for inference in production?

#48

Post by ein0p »

emu wrote: post #46
ein0p wrote: post #35

They aren’t really an alternative to anything. For one thing they’re now often slower on per-accelerator basis than NVIDIA stuff. They’re cheaper, of course, but because of disparity in performance you’ll need to estimate cost per flop on your own particular workload. They are also more difficult and slower to develop against, and SWE cost is always an issue if you don’t own a money printer like Google. Furthermore,…

> Such capability can’t even be contemplated on the TPU side because you basically get a black box. I'll just leave this here: https://jax.readthedocs.io/en/latest/pallas/index.html

Pallas is very new. Given how difficult these things are to debug and how half assed the XLA tooling generally is, I’d give it at least another year, if not two, before I touch it for anything prod related.

Re: Who uses Google TPUs for inference in production?

#49

Post by gperkins978 »

I tried to use a Google Coral. I have no idea how to make it work. I could follow a tutorial using tensorflow. I could not figure out how to use for anything else. Is there some way to run CUDA stuff on it? I always assumed it required someone with actual skills (not me). I have used CUDA stuff before, but more for mass calculation and simulation (for financial stuff). It is great when it works. I worked at a shop that had these Xeon Phi systems that worked great, but I had no clue how, and it only worked with their pre-canned tools.

Just as an example, over a decade ago I replaced a few cases filled with racks and a SAN that made up a compute cluster with one box (plus SAN) and a backup box (both boxes were basically the same in case one failed), but basically like dozens of servers were replaced by a two CPU box with a couple Tesla cards (probably one A100 later). The entire model had to be re-written, but it was not that bad. I wanted to do with AMD cards, but there was no easy way.

I would also say that modern networked has made all kinds of stuff more interesting (also lining Nvidia's pockets). Those TPU's do not make sense to me. I have no idea how to use them. They should release their version of CUDA.

Re: Who uses Google TPUs for inference in production?

#50

Post by fnord77 »

TwentyPosts wrote: post #45
fnord77 wrote: post #37

Earlier quoted context omitted.

> Google has been doing AI before any other company even thought about it This not even remotely true. SRI was working on AI in various forms long before google existed

Who or what is SRI?

next to NASA, probably the most innovative organization in human history

https://www.sri.com/timeline-of-innovation/

Post reply on HN