Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

Post reply on HN
Print view View original

271–280 of 355 posts

Re: We reduced our server costs by moving away from AWS

#271

Post by jiveturkey »

MajimasEyepatch wrote: post #125
mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

You're not including reliability and availability projections, and the intangible cost of transferable skills wrt infrastructure. (ability to hire sufficiently skilled people to run it)

Re: We reduced our server costs by moving away from AWS

#272

Post by agentultra »

moduspol wrote: post #214
boltzmann-brain wrote: post #187

Earlier quoted context omitted.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is dra…

How many applications need infinitely scalable distributed object storage?

I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.

But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.

Re: We reduced our server costs by moving away from AWS

#273

Post by Melatonic »

floatinglotus wrote: post #45

This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. My new startup is focused on helping application owners repatriate their workloads into their own infrastructure. Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you woul…

Very interested - you need people?

Re: We reduced our server costs by moving away from AWS

#274

Post by icedchai »

dijit wrote: post #253
icedchai wrote: post #241

Earlier quoted context omitted.

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians wor…

I'm sure we all had different experiences with old school sysadmins. The ones I'm familiar with (early 90's, ISP industry) could do shell scripting and some perl. C was way out of their skill sets, except for a bit of copy-and-paste.

The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.

Re: We reduced our server costs by moving away from AWS

#275

Post by pclmulqdq »

pwinnski wrote: post #255
pclmulqdq wrote: post #182

Earlier quoted context omitted.

In my experience, it is almost always not cheaper to use AWS services, unless your workload is exceptionally bursty in an unpredictable way or fits entirely in the free tier. Pretty much nothing at AWS is a loss-leader except the free tier. Also, if your workload is small and you don't need to hire cloud ops folks, you can come out ahead on TCO.

So we have gone from a categorical "not cheaper" to "almost always not cheaper," except for two circumstances, no, three. Progress! At my last company, providing SaaS for the education market, moving from a datacenter to AWS saved almost 70% year-over-year. In the datacenter, we ran machines to cover our peak load, which only happened a few times a year. In AWS, we scaled way down and auto-scaled up during those peak…

It sounds like you made a good decision, and yes, it sounds like your entire market (particularly the parts around exam hosting) is exceptionally bursty. I worked with some folks at a university on some of their IT systems, and they hit huge traffic spikes (literally 1000x their base load) around course registration and final exams. Their solution was to put 100 gbps NICs on their Oracle server...

I don't know how big your peak was compared to your average load, but if it was anything like that, serverless was a great call. Buying enough hardware for the bar exam and idling it 99.5% of the time sounds incredibly wasteful. However, this is an exceptionally bursty workload.

Most services do not have this level of traffic variability in such a way that a CDN can't handle it for you. 10x peak to trough variability (after your CDN) is fairly common, but still considered bursty, and in that case, AWS serverless still doesn't look great compared to DO droplets. Many services have daily or weekly cycles (without events like the bar exam), and run analytics workloads in their off hours.

Re: We reduced our server costs by moving away from AWS

#276

Post by icedchai »

jjav wrote: post #263
icedchai wrote: post #241

Earlier quoted context omitted.

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

> Most old school sysadmins were doing everything manually with relatively little automation. That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there. The growth of perl, for example, was in great part from the sysadmin community automating everything.

Most of the scripts I was familiar with from that time were one-offs. The code wasn't very reusable. They were automating a task on one machine. Today's "DevOps" are automating things across N machines. It is a matter of scale.

Re: We reduced our server costs by moving away from AWS

#277

Post by jjav »

joshstrange wrote: post #140
shudza wrote: post #112

Earlier quoted context omitted.

So what should you use for let say, a backend API service? Don't tell me beanstalk/fargate/etc, because they're actually more expensive.

Lambda is an option, it's been a very attractive one for me.

AWS lambda is super cheap at tiny scale. But if you get into any real constant load, it is way more expensive than a VM.

The other drawback of lambda is the flip side of having the server opaquely managed by AWS. It is so opaque you can't debug anything. At one startup we had a weird connectivity issue from lambda to RDS but it was impossible to diagnose given the lack of access so it went on for months.

Had it been running in a VM, I could've diagnosed that within an hour with tcpdump and bpf et.al.

Re: We reduced our server costs by moving away from AWS

#278

Post by dijit »

icedchai wrote: post #274
dijit wrote: post #253

Earlier quoted context omitted.

You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians wor…

I'm sure we all had different experiences with old school sysadmins. The ones I'm familiar with (early 90's, ISP industry) could do shell scripting and some perl. C was way out of their skill sets, except for a bit of copy-and-paste. The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.

I’m biased. I worked in teams with such people basically my whole career.

The pay increased, the titles and tools changed but the mentality didn’t.

People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are.

Sounds like they were automating though.

“Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.

Re: We reduced our server costs by moving away from AWS

#279

Post by GauntletWizard »

Melatonic wrote: post #262
hintymad wrote: post #218

Earlier quoted context omitted.

People don't consider productivity? Maybe things have gotten a lot better in the industry now. Otherwise, to rehash an older comment on HN: I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was ba…

That also just sounds like Uber had hired crap talent....

And crap management to lead them. Uber hired plenty of people smart enough to do better, but let the crap take the reins and management failed to lead on anything.

Re: We reduced our server costs by moving away from AWS

#280

Post by jjav »

pwinnski wrote: post #265
60secs wrote: post #180

Earlier quoted context omitted.

If you need to hire 5-10 engineers to save $1MM, you didn't save anything.

Right, which is why I can't imagine moving from AWS to self-hosting. Having to hire a full set of Ops people to manage what AWS is managing for me would hurt.

How are you running AWS without people?

Our AWS DevOps team isn't any smaller than they'd be if they were running a handful of servers in a colocation facility.

Post reply on HN