Live data from Hacker News

Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

cloud.google.com

11–20 of 28 posts

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#11
post #10
post #9

Earlier quoted context omitted.

They lost their way or HN readers are out of touch?

Google Drive losing customer data is which one? Context: https://www.theregister.com/2023/11/27/google_drive_files_di...

Actually if you think that Register report supports your first sentence you are perfectly demonstrating that lack of critical thinking that permeates this site.

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#12
Fantastic blog! In your post, you highlighted the use of Cloud TPU v5e with GKE for AI inference. How does this setup maintain high performance while managing costs, especially in high-demand scenarios like real-time data processing or live interactions?

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#13

Fantastic blog! In your post, you highlighted the use of Cloud TPU v5e with GKE for AI inference. How does this setup maintain high performance while managing costs, especially in high-demand scenarios like real-time data processing or live interactions?

Thank you! Regarding performance, TPU v5e which was benchmarked in the MLPerf results and showed impressive performance/dollar, see https://cloud.google.com/blog/products/compute/performance-p... which has more details. Combined with k8s & GKE, TPU v5e workloads can leverage auto-scaling, for example setting up autoscaling based on traffic, so workloads can scale down rapidly when not in use, increasing cost efficiency.

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#15
AFAIK Google is in the middle of massive infrastructure investment in hardware which is of similar performance to Nvidia, but they will own the whole stack. They will also be able to deploy an order of magnitude more compute than openAI to LLMs in 2024/25.

LLM tech also seems to be easily swapped between providers. So my view is that the value proposition is in the hardware designers and manufacturers (Nvidia, tsmc, Google).

Why are more people not excited by what is happening at Google? What am I missing?

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#16

AI is a big deal this year. Google is one of the biggest tech companies. Yet a post about AI at Google gets... No comments at all. Back in 2005, this would have been the most talked about news of the day. Now, nobody cares what Googles up to. They lost their way.

My vague read from before was that TPU4 has been out for a long long time & was basically near impossible to get time in unless you could bid super high.

Not my field & this could be super wrong. But my skepticism that a TPU5e launch was going to help anyone but some niche well connected other big players has been very high. I'm interested from an academic sense, and I love work like the Keras 3.0 release that makes me much more universal/portable, such that hardware like this or others can burn f-ing metal. But I don't have much trust this will actually be available at a reasonable price/with any kind of availability. https://news.ycombinator.com/item?id=38446353 https://keras.io/keras_3/

I'd love to be wrong & have this be actually available at a good price to change what is possible for people, & still interesting. Your call out is right. There's no lightning rod here, nothing to be hyoer-critical of, and few step outside the comfort zone of CUDA have anywhere near enough idea how to assess. This is an unknown on the radar for most.

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#17
post #15

AFAIK Google is in the middle of massive infrastructure investment in hardware which is of similar performance to Nvidia, but they will own the whole stack. They will also be able to deploy an order of magnitude more compute than openAI to LLMs in 2024/25. LLM tech also seems to be easily swapped between providers. So my view is that the value proposition is in the hardware designers and manufacturers (Nvidia, tsmc,…

To the end user, everything at Google has become shit or not changed at all. I am sure GCS is great but most people don't use it. What they do see is search becoming more ads than content and search not even including any relevant search terms in "here are your 1000000 results". I actually go to yandex and bing if I want to search for a specific search term.

Let's not talk about friends losing all access to their accounts, forever.

I switched a whole firm off GCS when their troubleshooting by 'senior' techs was just copy and pasted boilerplate too many times. The way I set things up was cloud agnostic at the time. Not something that is possible at scale.

Can anyone else be trusted? Probably not.

Can you trust someone on the crack-pipe full of advertising dollars?

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#18
post #11
post #10

Earlier quoted context omitted.

Google Drive losing customer data is which one? Context: https://www.theregister.com/2023/11/27/google_drive_files_di...

Actually if you think that Register report supports your first sentence you are perfectly demonstrating that lack of critical thinking that permeates this site.

How do you see the linked issue?

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#19
post #4

AI is a big deal this year. Google is one of the biggest tech companies. Yet a post about AI at Google gets... No comments at all. Back in 2005, this would have been the most talked about news of the day. Now, nobody cares what Googles up to. They lost their way.

Google, structurally, sucks at anything but sucking down Ad money. A good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time. As cool as LLM capabilities are the thing that delights me most about them is I can get high quality (YMMV) answers from the internet without all the godforsaken ad bullshit and wonky websites made only as mazes to keep users in ad gardens. Googl…

LLMs are great at historical data. They however still suck at being kept up to date. I would expect that someone (if not already has) will find a cost efficient way to continuously index new data, but it would be interesting to see how it stacks up with the computing power needed for modern search index pipelines.

Re: Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

#20

AI is a big deal this year. Google is one of the biggest tech companies. Yet a post about AI at Google gets... No comments at all. Back in 2005, this would have been the most talked about news of the day. Now, nobody cares what Googles up to. They lost their way.

[deleted]
Post reply on HN