Cloud services like AWS or Google Cloud Platform may be the wrong choice

I am currently in the middle of setting up AWS and that decision graph made me chuckle because it resonates quite deeply.

I need to do GPU inference but I don't want to run the machine 24x7. I may use it for about 4 hours per day at best. Lambda doesn't offer GPUs and neither does ECS+Fargate.

It seems like I could setup an endpoint using Sagemaker and then destroy it when no longer needed, and automate all of this but it feels quite messy.

The other route is perhaps I can launch an instance every day with ECS and then get rid of it.

All these routes seem quite inefficient. There seems to be something called Elastic inference where I can provision the right amount of GPU resources - but it seems like I'll need a spare EC2 instance to do that if I'm not mistaken, which is not ideal either.

I guess all this stems from the fact that there is no straightforward virtualization for GPU workloads and so they have to provision them 1:1 which currently they are not equipped to do.

Has anyone run into a similar problem and found a more elegant solution? All of the above are very messy. Is there some obvious choice I am missing?

Agree, we bootstrapped a business from cents a month to $4 or $5 now, there's no maintenance, and I know if we get mentioned on Oprah - we'll be able to cope with a blip thousands of signups a second. I know how I'd run the system on our own hardware but can put off that decision to when (or if) we need it.

I spent a few evenings going the ECS route and the complexity and cost of just a single api available to the internet and a database kept ballooning to way more than what I wanted to pay for and deal with.

Fargate, image repo, NAT gateway, I don't even remember all the nonsense at this point but it was ridiculous.

I took a look at a few alternatives and DigitalOcean's App service was night and day easier, faster, and cheaper to deal with.

> what's weird though is how many mega-corps are going away from Custom Hardware in Custom Built DC towards Cloud.

Why is it surprising? Building and maintaining custom data centers is a big, slow business initiative. It takes months to years of forecasting to get the data center buildout to match the business needs, as opposed to the extreme flexibility of using a cloud provider.

> There's also something to be said for buying a VPS or a Colo machine, making sure it's backed up and dealing with the 9's that you get from that machine on it's own. I am routinely surprised by how far a single node machine will get you.

For personal projects this is exactly what I do. It’s great until something goes wrong with that one machine or VPS.

But it’s not really a good option for any business that needs consistent operations and uptime. Years ago I worked at a company that tried to self-host some of their collaboration tools on a VPS to save money over the cloud-hosted versions. When the server went down it stalled productivity for a day while the team restored a backup, with another week of confusion as we tried to find all of the things that were lost between the last backup and when the server went down.

When someone did the rough estimations on how much it cost to pay everyone’s salaries for that day of lost productivity, the number was far higher than the trivial cost savings we got from self-hosting. We also had a constant background burden on someone internally to maintain and monitor the server, plus the burden of them being on call. Often, moving to cloud anything can be a huge load off the company’s back.

If your GPU inference can run on an Intel integrated GPU, you could rent a dedicated server from OVH for ~$130 a month and use the integrated GPU on that. I don't know about affordable dedicated servers from mature providers with Nvidia GPUs.