Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Designing a scalable API on AWS spot instances

blog.adapty.io

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Designing a scalable API on AWS spot instances

#21

Post by [deleted] »

dijit wrote: post #8
PetahNZ wrote: post #5

Earlier quoted context omitted.

Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?

The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware. I'm not sure where the idea came from that this is hard to do. To answer your questions in order: 1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy. I'm not saying it's black and white; but I hope…

[deleted]

Re: Designing a scalable API on AWS spot instances

#22

Post by yellow_mixer »

marcinzm wrote: post #19
makkesk8 wrote: post #3

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs" This is a recurring topic here on HN and it boggles me and makes me wonder if people know that there are other platforms than aws, azure and google cloud out there that are very capable and much much cheaper. Unless any of the big 3 has a feature or certification you need I don't see any reason to use them at all due to the insane…

>So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point? Because it takes X hours of effort to cut costs and using another provider would have required Y hours of effort where Y >> X. Y may be greater due to reliability issues, missing features that you need to build yourself, training costs of new employees, etc. edit: Also X is paid once you're succeeding, Y has to be…

It’s also much easier to hire engineers to develop and maintain theses platforms.

Re: Designing a scalable API on AWS spot instances

#23

Post by dijit »

dijit wrote: post #8
PetahNZ wrote: post #5

Earlier quoted context omitted.

Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?

The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware. I'm not sure where the idea came from that this is hard to do. To answer your questions in order: 1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy. I'm not saying it's black and white; but I hope…

Someone made a good reply to this and as I was replying to that they deleted it, so I'll copy it here:

----

> Have you actually done a cost-benefit analysis on some of these solutions?

Yes, I even gave a talk at google in stockholm about it.

For my use-case, hybrid was best, with no cloud lock-in aside from Google Storage Buckets (which can be replaced) but I went into detail about that in the talk.

> Take your Riak / S3 plugin. What do your servers cost to run that cluster?

Depends a lot, don't you think?

> How much time do you spend managing it?

Depends again, if it's anything like my elasticseach clusters then about 2-man hrs/mo.

> How do you test your backups?

Continuously, and with alerting.. and, you should be doing this anyway.

> Are you going to target the same SLOs for durability that S3 offers?

Depends on the business, the whole point of SLO is that you pay in what it's worth to the business.

> Do you run multi-data center for high availability?

Depends on SLO.

> In many cases the cheaper or self-hosted solutions have costs that you aren't accounting for.

Yes, physical machines often need some hand-holding, VPS's can have brown outs, but this is true in AWS's EC2 anyway.

Ultimately, this is where the cost increase will be.. but defining it is important, I've deployed cloud and physical (as stated) and it's true that physical machines are not as problem-free as our GCE ones- but we pay about 50% less than the GCE equiv instances, so it's "worth" spending time automating the unpredictable.

> Sometimes that's fine, but "just run it yourself" is as worthless as saying "just ship it to AWS" unless you actually think through the impact.

This is kind of the main point I always make.. understand your trade-offs, don't buy into proprietary tech. Cloud is a fantastic way to prototype and bootstrap but it's /usually/ better to have a migration plan to optimise costs in the future.

If you fail to take that into account then I don't have sympathy for you, because you put the project at risk. Financial in-viability is a risk.

Re: Designing a scalable API on AWS spot instances

#24

Post by jonatron »

csunbird wrote: post #15
jorams wrote: post #4

Earlier quoted context omitted.

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Stor…

is it that bad? we are thinking about bootstrapping a project, which will use spaces for file storage and we will be using them frequently.

I tried it for a small personal project, just for somewhere to put some small images. On pages of about 30 images, usually one of them just wouldn't load. It didn't look reliable to me either.

Re: Designing a scalable API on AWS spot instances

#25

Post by dijit »

saddlerustle wrote: post #10
jorams wrote: post #4

Earlier quoted context omitted.

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Stor…

This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky. Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.

AWS S3 buckets have a limit of 5gb/s access to a single bucket.

Unless that limitation has changed in the last couple of years; I can easily make a system beat that, if that's the requirement.. and ultimately it does come down to understanding requirements. :\

I think generally people forget that cloud is just computers too, it's really not anything special, and amazon/google/microsoft are solving the general case (and, doing so well, actually) but this comes at a high premium.

Re: Designing a scalable API on AWS spot instances

#26

Post by fulafel »

saddlerustle wrote: post #10
jorams wrote: post #4

Earlier quoted context omitted.

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Stor…

This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky. Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.

There's options outside SaaS object storage though, eg running your own MinIO or Ceph, + DAS (or SAN) at your colo. Yes it's some hassle but if you have significant expenses from SaaS it might be worth checking out.

Re: Designing a scalable API on AWS spot instances

#27

Post by l33tman »

makkesk8 wrote: post #3

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs" This is a recurring topic here on HN and it boggles me and makes me wonder if people know that there are other platforms than aws, azure and google cloud out there that are very capable and much much cheaper. Unless any of the big 3 has a feature or certification you need I don't see any reason to use them at all due to the insane…

I'm sure there are other services that are cheaper in some of the areas, but at least the argument about "insane complexity" is moot; if you've used AWS before it's no problem to set something up from scratch. I've done it numerous times and it's a one day job to setup most of the infrastructure required with VPCs, IAM users, policies, databases, scaling groups, storage configs etc.. That part of it is not really an argument to switch at least.

Re: Designing a scalable API on AWS spot instances

#28

Post by l33tman »

ollyculverhouse wrote: post #7
ocdnix wrote: post #2

Turns out this is about EC2 spot instances for ECS. How would it compare to ECS Fargate spot these days? I'm also missing a discussion about designing for interruption, either by not keeping state, or by being able to shed state quickly, to be picked up by other instances. Also, if you set up EC2 spot with a launch template or ASG with very differently-sized instance types (to reduce risk of running out), is there a…

AFAIK with Fargate a lot of this is handled for you, as long as you have the auto scaling group. We have this setup with two capacity providers (FARGATE_SPOT and FARGATE) with a 75/25% split, meaning that even if there are no spot instances available we will still be up. The benefit of Fargate being that we don't need to care if certain instance sizes are not available as that is handled by AWS.

Cool, when fargate launched they didn't have a spot possibility (AFAIK) and since we run ECS on Spot instances it would just be a massive increase in cost to switch to FG, but if it now can use underlying spot instances, it might be worth looking at again..

Re: Designing a scalable API on AWS spot instances

#29

Post by owenmarshall »

dijit wrote: post #23
dijit wrote: post #8

Earlier quoted context omitted.

The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware. I'm not sure where the idea came from that this is hard to do. To answer your questions in order: 1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy. I'm not saying it's black and white; but I hope…

Someone made a good reply to this and as I was replying to that they deleted it, so I'll copy it here: ---- > Have you actually done a cost-benefit analysis on some of these solutions? Yes, I even gave a talk at google in stockholm about it. For my use-case, hybrid was best, with no cloud lock-in aside from Google Storage Buckets (which can be replaced) but I went into detail about that in the talk. > Take your Riak…

That was me ;) I deleted it because I read your second post and realized I'd completely misread you and actually agree almost in entirety.

> understand your trade-offs, don't buy into proprietary tech. Cloud is a fantastic way to prototype and bootstrap but it's /usually/ better to have a migration plan to optimise costs in the future.

Preach.

Re: Designing a scalable API on AWS spot instances

#30

Post by jakozaur »

For service type workloads (e.g API service with 99.99% uptime SLA) we keep comparing on-demand vs. spot.

In reality, you would like to compare Reserved Instances as you can get 60% discount.

So in us-east-1:

- spot costs: 35-43% of on-demand

- RI 1 year standard: 60% of on-demand

- RI 3 year convertible: 46% of on-demand

So if you have some base load that you can commit to running for 3 years, the price gets often at spot range while not having to worry about losing capacity.

In-reality sometimes combining reserved for some base capacity 60% + 40% spot for spiky seems to be the winning combination for many companies.

Post reply on HN