61–70 of 191 posts
Earlier quoted context omitted.
Not to mention, if it's an ML workload, you'll also have to factor in downloading the weights and loading them into memory, which can double that time or more.
According to the press release, "we achieved an impressive Time-to-First-Token of approximately 19 seconds for a gemma3:4b model" Imagine, you have a very small weak model, and you have to wait 20 seconds for your request.
Earlier quoted context omitted.
Or go P2P with Vast.ai, cheapest A100 right now is a setup with 2x A100 for $0.8/hour (so $0.4 per A100). Not affiliated with them, but mostly happy user. Be vary of network speeds though, some hosts are clearly on shared bandwidth and reported numbers don't always line up with reality, which kind of sucks when you're trying to shuffle around 100GB of data.
You really need NVL for some performance.
once this bubble pops we are going to have some serious albeit high-latency hardware
Earlier quoted context omitted.
Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.
I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.
i wonder what all this hype-driven overcapacity will be used for by future generations. once this bubble pops we are going to have some serious albeit high-latency hardware
All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
Cloud Run is great but no billing limits is too scary. No idea why they don't address this. They must know if they support individuals we'll eventually leave our saases there.
Earlier quoted context omitted.
Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.
I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.
All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
Not providing a cap on spending is a major flaw of GCP for individuals / small projects. With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)
Feature for Google's profits.
Earlier quoted context omitted.
Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :( I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.
I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker. He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases. His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.
I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to prioritize this feature, since they’re not spending very much relative to customers who wouldn’t want this feature.
Maybe if there weren’t more feature requests to get prioritized this might happen, but the reality is that there are always more feature requests than time to implement them, and a feature request used almost exclusively by the smallest dollar customers will always lose to a feature for big-dollar customers.
i wonder what all this hype-driven overcapacity will be used for by future generations. once this bubble pops we are going to have some serious albeit high-latency hardware
Crunching really large amounts of numbers has always been useful. And that's all this really is. Running weather simulations, Advanced math problems, Complicated engineering simulations. The space of possible uses is incredibly wide.