Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

Post reply on HN
Print view View original

141–150 of 355 posts

Re: We reduced our server costs by moving away from AWS

#141

Post by pclmulqdq »

etothepii wrote: post #114
datalopers wrote: post #97

Earlier quoted context omitted.

EC2 is vastly more cost efficient than using all of those managed services unless you only have tiny on-demand usage scenarios.

Or any kind of demand spikes. If you are using lambda and suddenly need to run a million lambda you just can. If you want to have the ec2 support to run a million lambda at some time you are going to pay for a lot of sleeping computers.

If your spikes go 5x over your base load, EC2 is still cheaper. Even if your load spikes are 10-20x over your base load, EC2 can still be cheaper. Past that, lambda is your better bet. If you are comparing to Hetzner or OVHCloud, you those numbers are 25x (definitely cheaper) and 50-100x (possibly cheaper).

Re: We reduced our server costs by moving away from AWS

#142

Post by theptip »

throwaway858 wrote: post #120
alberth wrote: post #35

Dedicated hosting providers. I'm so amazed that somehow people completely forget that for literally decades, web host provided dedicated hosting options at fantastic prices. Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited. But large numbers of web host today allow you to programmatically spin up a d…

The one big thing missing from dedicated hosts is an S3 equivalent. Sure, you can get a huge hard drive for cheap but this will not have the durability requirements for storing your precious data. And if you try to use AWS just for S3 then you will pay a lot extra for the bandwidth charges of bringing the data from S3 to your server (something that is free if you were to use EC2 or other AWS services).

Right -- that's the one implementation detail in the OP that was interesting. It sounds like they ultimately used MinIO to replace S3. I've seen people use Ceph, but it's apparently a nightmare to operate a Ceph cluster. If you're on k8s I think the "cloud native" way might be Rook, haven't looked into that. Anyway, running an object store is painful.

Their notes here are a bit vague:

> When the migration reached mid-June, we had 300 servers running very smoothly with a total 200 million cached pages. We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3.

> We broke the online migration into four steps, each a week or two apart. After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio. When the writes to S3 had been stopped completely, Prerender saved $200 a day on S3 API costs and signaled we were ready to start deleting data already cached in our Cassandra cluster.

> However, the big reveal came at the end of this phase around June 24th. In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster. The daily cost of AWS was reduced to $1.1K per day, projecting to 35K per month, and the new servers’ monthly recurring cost was estimated to be around 14K.

It says (briefly, in passing) that they used Cassandra to implement the S3 API for their nodes, but maybe just to replicate the S3 API that they were previously using? That's an interesting choice I'd not heard of before. Perhaps all of their individual files are quite small?

Then they moved to MinIO, which would be the S3 equivalent that you are looking for.

Re: We reduced our server costs by moving away from AWS

#143

Post by pclmulqdq »

sparker72678 wrote: post #71
macinjosh wrote: post #65

Earlier quoted context omitted.

What I don't understand is why anyone would spend, precious and expensive engineering resources on coding to a proprietary API owned by one of the largest companies on the planet to whom you are nothing. Ok so you save money on the monthly bill but what happens when Amazon decides they want to enter your market? What happens if they decide your service is too controversial and they kick you off? If you were deploying…

The alternatives are tremendously more expensive in both time and money (and I’m thinking engineer salaries etc. here, not just monthly service costs). For most businesses the savings in money and time are worth the risks you pointed out here. If those risks are too high for you (and fair enough if they are), you’ll pay one way or another to avoid them.

Is your cloud ops department free? I have personally seen that amazing sysadmins are a lot cheaper than passable cloud ops people.

Re: We reduced our server costs by moving away from AWS

#145

Post by acdha »

datalopers wrote: post #97
joshstrange wrote: post #48

They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce yo…

EC2 is vastly more cost efficient than using all of those managed services unless you only have tiny on-demand usage scenarios.

That's definitely not true for “all of those managed services” — you can't replicate CloudFront on EC2, for example, and it'd be extremely unlikely that you could build your own S3 replacement on top of EC2 storage without spending considerably more on operations than you save. On-premise storage is cheaper but you're still going to struggle to see a cost savings unless you either don't care about reliability or availability and/or buy your data by the petabyte.

For the other services, it's still going to depend heavily on your usage, staff time, and operational efficiencies. For example, if you replace SQS with RabbitMQ you need to manage multiple EC2 servers for reliability but those servers might be significantly underutilized based on your traffic levels. Whether or not you save money depends on how much you pay your operators and how many messages you use: if you use less than the free tier's million messages per month, there's no way to beat it. Each million requests costs $0.40 or less, so if you pay your ops person $50k annually (haha) you'd need to be somewhat over 120 million messages per month to pay for them to spend a single hour on O&M even before factoring in your EC2 usage.

Re: We reduced our server costs by moving away from AWS

#146

Post by P5fRxh5kUvp2th »

unregistereddev wrote: post #80
P5fRxh5kUvp2th wrote: post #33

Earlier quoted context omitted.

Any developer that can't setup a server has fundamental gaps in their skillset.

On a trivial level I agree, but only on a trivial level. Most developers will not set up a server in the most secure, maintainable manner. They're going to miss important things: - Disable password-based auth for ssh (require key-based auth) - Enable fail2ban or similar to slow down brute-force login attempts - Configure firewall - Install monitoring tools, antivirus, possibly backup daemons, etc - Setup a sane swapf…

I have a rant here about security being mostly bullshit. Not even sysadmins can setup a server "in the most secure, maintainable manner".

You don't hire a dedicated sys admin because they know some voodoo magic developers don't, you hire a sys admin because they have the _TIME_ to watch for security updates. And you _PAY_ a company like RedHat to ease that burden.

A dedicated sysadmin will typically do a better job than a non-dedicated developer, but if that developer cannot do a passable job, that's a gap in their skillset.

I may not be a network engineer, but I damned well understand DNS, MPLS, networking, gateways, and the like. Any developer who doesn't has gaps.

Re: We reduced our server costs by moving away from AWS

#148

Post by hotpotamus »

blackoil wrote: post #119
onlyrealcuzzo wrote: post #105

Earlier quoted context omitted.

This is likely only true until they have enough lock-in on those cloud products to really turn the screws and extract maximum profits from people that are TOTALLY screwed by being completely locked into proprietary AWS services... This is basically the natural progression of SaaS...

This is often mentioned scenario but highly unlikely as impractical it is. First, AWS is already highly profitable, they aren't dumping where this step becomes a necessity. Also, any such move would dry up future growth from adding new customers, which is suicidal.

Is AWS profitable enough though?

Re: We reduced our server costs by moving away from AWS

#149

Post by spoiler »

twawaaay wrote: post #70
joshstrange wrote: post #48

They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce yo…

Yeah, you need to do calculation of what you are doing right now vs AWS. If you already have a mature team that is doing well optimising the environment, you have stable demand and no need to develop things rapidly then it is very likely that going cloud is poor choice. As to "cloud agnostic", don't believe this bullshit. In my opinion most projects fare much better by just letting go of this "cloud agnostic" and jus…

> As to "cloud agnostic", don't believe this bullshit.

A notable exception to this is of you have clients whose IT departments require you to work on specific cloud providers. Although, there's tools that help abstract deploying infrastructure to different cloud providers (eg terraform, pulumi), but they still require some familiarity with the providers. With all that said, overall I agree with your sentiment

Re: We reduced our server costs by moving away from AWS

#150

Post by 300bps »

shudza wrote: post #112
taylodl wrote: post #59

Earlier quoted context omitted.

Yes. If you use and manage AWS resources as you would your own on-prem resources then you're not going to have a good time. As soon as you think you need EC2 instances you need to re-think your architecture. You're probably not using AWS most effectively.

So what should you use for let say, a backend API service? Don't tell me beanstalk/fargate/etc, because they're actually more expensive.

Lambda, SQS and SNS.

That’s how just about everyone does backend API services with AWS. You’ll be shocked at how cheap, scalable and bullet-proof it can be.

Post reply on HN