101–110 of 134 posts
I‘m a CTO of a compute intensive AI SaaS company, so I can relate. One advice: speak to your AWS rep immediately. Get credits to redesign your system and keep you running. you can expect up to 7 digits in credits (for real!) and support for a year for free, they really want to help you in avoiding this.
You have to use Spot instances. Or as Google calls them - preemptible instances. These are upto 80% cheaper.
The caveat is that they can be killed anytime, so your infrastructure must be resumable.
Most likely you will need to do kubernetes. It's the only framework that supports GPU, integrates with spot instance providers and works with Ml platforms (using kubeflow)
Secondly, you can use AWS spot instances for GPUs which costs less.
Earlier quoted context omitted.
Also, consider physical servers from providers like Hetzner. These can be several times cheaper than EC2.
I was just looking at Hetzner yesterday, looking to host a HA Postgres setup. Their block storage volumes look interesting, but I couldn't find any information on performance guarantees, or even claims. Anyone have an idea about performance (IOPS or MB/s)?
Check his profile.
He has people, or knows of people that can can likely help with this. He's a CEO of a an Accelerator and is not a newb by any sense.
OR... he's using gamification to find someone to hire to actually help solve this problem. If that's the case... Bravo sir!
Email me lanec (at) hey (dot) com if you’d like to speak.
Last year I took a company spending $24k/month training visual AI and cut that down to $3,500/month with bare metal. I also helped them secure over $100k in cloud credits to cover the monthly costs until the transition could happen.
Training in the cloud is generally much more expensive than bare metal.
Also, if you can't afford to scale to new customers... stop? I'm sure it probably sucks, but like, does it suck more than having no runway? Seems like you'd be best served slowing things down and spending some time with AWS on cost optimization.
There aren't a lot of details to go off of here so I don't know what more advice to give.
Earlier quoted context omitted.
I never defended AWS as being the absolute best. I said high bills are almost always due to developers and not the cloud provider. Which you haven’t argued against. As I said I have examples of how Developers often cause large bills. And I explained why we can’t help with the OPs large bill. You’re saying that with ML there is absolutely 0 way to reduce costs on AWS which is absolute rubbish.
> I said high bills are almost always due to developers and not the cloud provider. I feel that's where you keep missing the whole point. Somehow you're stuck on thinking that an expensive service is not a problem if you can waste time micromanaging and constantly monitoring expenditures to shave off a bit of cost from the invoice. Yet, somehow you don't register in your universe the fact that there are services out…
And that for cases like MLing it’s impossible to optimise costs.
Got ya.
Earlier quoted context omitted.
> AWS/clouds aren't always the best solution for a problem. And when they aren’t always the best. It’s often because you don’t know what you’re doing. It’s too uncommon for people to over provision. Or go with too many services when they don’t need to. Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough…
Nothing is stopping you from applying all those optimizations to on-premise hardware, right? That is, I am not sure "public cloud, if you spend lots of effort to optimize it and ask devs to be careful, can be as cheap as a naive on-prem implementation where devs don't need to be careful" is an argument for public cloud.
I’m also not suggesting optimising and being careful is an argument for cloud. I’m saying that ruling out cloud is stupid. You can absolute have a Low cost solution perform very well on a cloud provider. The OP seems to think it’s not possible.
My pitch to help: you can probably replace the GPU-intensive ML model with some incredibly dumb linear model. The difference in accuracy/precision/recall/F1 score might only be a few percentage points, and the linear model training time will be lightning fast. There are enough libraries out there to make it painless in any language. It's unlikely that your users are going to notice the accuracy difference between the…