Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

Between its support for K8s, networking stack/infra, and TPUs/ML support we have a lot of huge well known tech companies knocking down our door.

But none of those things really matter as a differentiator for the average startup, scale up, or small business. So all in all, i’m not surprised the majority of HN engineers just aren’t excited by GCP.

I have no knowledge of Cloud’s long term strategy, but it wouldn’t surprise me if by leaning into its success in big tech, they are neglecting the startup scene in terms of marketing or building out new features.

I'd love contra-indicators. I'm not plugged in to this community nearly as muc has I am the rest of tech. But the general scuttlebutt I got was that access to TPU4 was quite competitive. It didn't seem to be like a cloud resource where you could just go use it: it took work and effort and time and many other people could butt in line ahead of you all the time to get TPUv4. Apologies for relaying hearsay, but if this is far better tech, it feels like Google is going to have to solve the scale-out concerns exponentially harder if TPUv5e is to actually* have an effect on most practitioners. It's my hope that these negative concerns were overstated; it's my hope that the situation is better. Does anyone have any idea what we could look at to suss how much of a concern this is?

I think you've also casually stumbled into another issue. To quote you:

> but they will own the whole stack.

This feels like the typical slant against Google, and I think often this is a disservice.

I personally attribute much better to Google. To me, Google has been one of the few companies that still has some of the intertwingularity in their ethos, that still has some Cluetrain Manifesto blood in their veins where they grok that the way to be a winning player is to bring a ton of selfless. But we keep seeing spin that makes Google look egocentric & self-biased & self-dealing, in so many ways. Any bad Google anywhere outshadows the rest of the good Google; Google is a massive goliath powered by the vastest ad market on the planet, but where-as most companies constantly focus on mining value of their ecosystem & imho Google is one of the rare few companies that groks that the existential stakes of creating open healthy accessible ecosystems. Imho they have almost never stacked the deck to be overlords (fuck you very very much MV3, you self-dealing app-store owners how casually rescind so many dimensions of freedom so easily for yourselves) but instead espoused a multi-polar world, one made up of many independent empowered factions. If anything, I think much of the scorn laid at Chrome's feet, for example, actually reflects on the rest of the world's inability to be as good, to serve as even remotely adequate patrons to the web: governments everywhere should be funding web development, web specs, web standards. There's just so few players & that's not Google's fault. The bread & butter of open information society is all too moldy & stale in most areas, too few players. Still, I've seen Google persist, try. I find it admirable. I see this in many ways, but the faults & criticisms are always such drastic lightning rods for anger.

No anger here yet, but: there's hardly a better example of Google trying to play well than today's Keras 3.0 announcement. 'Hello, we are Google, we've optimized the ever loving hell out of our existing machine learning platform to target a whole bunch of other environments, make them 2D-grid data & model parallel across a variety of frameworks targeting by far the widest range of hardwares. And so now, if you target Keras, any of these other platforms will work intercompatibly very well.' Once again, Google steps in the ring to carry water for the world, to make practioner's lives much easier. Sure, this enables TPUv5e to be easily targetted, via a variety of different code-styles; that is self interested. But it also enables a degree of portability out of TPU & across systems that seems magical, that is extremely late bound, that permits permits permits. I'm harping on just a small bit you said, apologies, but in contrast to your 'own the whole stack', it feels like Google is the only player trying to make sure no one owns the stack, and it seems like they're doing great at it. TPUv5e happens to be one option these actions bolster. https://keras.io/keras_3/ https://news.ycombinator.com/item?id=38446353

Why do these LLM startups have such large investments and such large spending? It's because the infrastructure costs of a GenAI startup is huge. That's most of the money that's changing hands. So, from the VCs down to accelerator platforms cost efficiency is a huge decision factor on all of this.