Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

Post reply on HN
Print view View original

331–340 of 355 posts

Re: We reduced our server costs by moving away from AWS

#331

Post by elcomet »

P5fRxh5kUvp2th wrote: post #37
rossdavidh wrote: post #7

Earlier quoted context omitted.

I think some of this is due to the presence (until just recently) of abundant VC money, encouraging tech companies to trade of money for time by outsourcing their server admin (or part of it, anyway). It's one less thing you have to hire for and spend time getting good corporate policies and procedures on. But, now that the emphasis seems to have shifted from growth-at-all-costs to where-can-we-cut-costs, I think the…

imo this is the right takeaway. You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious. It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious. I suspect most developers don't find it obvious because they've never dealt with actual ser…

Again, you seem to not take into account the developer/admin time. In your example with oil, if you track your time, it's gonna be certainly more expensive than going to the garage.

And also of course, it allows you to go faster, which can have a huge value

Re: We reduced our server costs by moving away from AWS

#332

Post by jen20 »

dijit wrote: post #253
icedchai wrote: post #241

Earlier quoted context omitted.

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians wor…

> Terraform was written by a sysadmin

I’m with you on most of your post, but this is straight up wrong.

Re: We reduced our server costs by moving away from AWS

#333

Post by dijit »

jen20 wrote: post #332
dijit wrote: post #253

Earlier quoted context omitted.

You’re quite mistaken on the first point. Most automation that you know of as “devops tools” are borne from sysadmins. Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too. It is revisionist to think sysadmins were not automating their jobs. Old school sysadmins used to know C, bash and Perl. New school devops just traded that for a handful of DSLs and Python. I’m sure some technicians wor…

> Terraform was written by a sysadmin I’m with you on most of your post, but this is straight up wrong.

The original author of terraform (Mitchel Hashimoto) was not an Operations Engineer before writing Terraform?

https://www.linkedin.com/in/mitchellh/

Or are you saying that he's not the author? Commits seem to suggest otherwise.

  commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b
  Author: Mitchell Hashimoto 
  Date:   Fri May 23 11:03:38 2014 -0700

     config: better error message

  commit 089822a36f0a5e4c583ab401b7496b48bfa31d65
  Author: Mitchell Hashimoto 
  Date:   Fri May 23 10:52:19 2014 -0700

     config: some comments

  commit ec3f72703c82f50787ef86498d856a5798900ad8
  Author: Mitchell Hashimoto 
  Date:   Thu May 22 16:56:28 2014 -0700

     Initial work on config

  commit 649cf336e86da10910f6e50ea3cfaad507a2e336
  Author: Mitchell Hashimoto 
  Date:   Wed May 21 16:28:53 2014 -0700

     Initial commit

Re: We reduced our server costs by moving away from AWS

#334

Post by arinlen »

dijit wrote: post #333
jen20 wrote: post #332

Earlier quoted context omitted.

> Terraform was written by a sysadmin I’m with you on most of your post, but this is straight up wrong.

The original author of terraform (Mitchel Hashimoto) was not an Operations Engineer before writing Terraform? https://www.linkedin.com/in/mitchellh/ Or are you saying that he's not the author? Commits seem to suggest otherwise. commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b Author: Mitchell Hashimoto Date: Fri May 23 11:03:38 2014 -0700 config: better error message commit 089822a36f0a5e4c583ab401b7496b48bfa31d65 Aut…

> (...) an Operations Engineer (...)

An operations engineer and a sysadmin are two entirely different roles.

A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.

Re: We reduced our server costs by moving away from AWS

#335

Post by dijit »

arinlen wrote: post #334
dijit wrote: post #333

Earlier quoted context omitted.

The original author of terraform (Mitchel Hashimoto) was not an Operations Engineer before writing Terraform? https://www.linkedin.com/in/mitchellh/ Or are you saying that he's not the author? Commits seem to suggest otherwise. commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b Author: Mitchell Hashimoto Date: Fri May 23 11:03:38 2014 -0700 config: better error message commit 089822a36f0a5e4c583ab401b7496b48bfa31d65 Aut…

> (...) an Operations Engineer (...) An operations engineer and a sysadmin are two entirely different roles. A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.

You're conflating an IT Technician and a Sysadmin.

Sysadmins automate their work. Operations Engineer is just another title for sysadmin.

Systems Engineer is also another title for the same thing.

Platform Engineering (depending on where you are) is yet another variation.

If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.

Re: We reduced our server costs by moving away from AWS

#336

Post by agentultra »

boltzmann-brain wrote: post #325
agentultra wrote: post #272

Earlier quoted context omitted.

How many applications need infinitely scalable distributed object storage? I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data. But those were outliers. All of the large and small enterprises outside of that could f…

You can hardly make quantitative statements about how many applications need this or that. It's not like we're in the 60s and IBM can tell that they'll need to make exactly seven of a certain computer. I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amo…

That does seem to be the common trend. If you generate that much data you tend to host on premises because the network becomes the bottleneck. It becomes important to place compute as close to the data as possible.

Re: We reduced our server costs by moving away from AWS

#337

Post by pwinnski »

pclmulqdq wrote: post #275
pwinnski wrote: post #255

Earlier quoted context omitted.

So we have gone from a categorical "not cheaper" to "almost always not cheaper," except for two circumstances, no, three. Progress! At my last company, providing SaaS for the education market, moving from a datacenter to AWS saved almost 70% year-over-year. In the datacenter, we ran machines to cover our peak load, which only happened a few times a year. In AWS, we scaled way down and auto-scaled up during those peak…

It sounds like you made a good decision, and yes, it sounds like your entire market (particularly the parts around exam hosting) is exceptionally bursty. I worked with some folks at a university on some of their IT systems, and they hit huge traffic spikes (literally 1000x their base load) around course registration and final exams. Their solution was to put 100 gbps NICs on their Oracle server... I don't know how bi…

My suggestion to you, then, is this: instead of making blanket statements and allowing for a teeny-tiny exception for edge case, try to remember that the market is FULL of those edge cases. Almost the entire education market, billions of dollars of revenue, fits into what you described as "exceptionally bursty" and didn't even include in your first blanket dismissal. I suspect education isn't the only market that surprises you with its burstiness.

We frequently see links at HN that fall over and die after they're linked. On the one hand, that's usually because they've misconfigured a personal blog and lack caching. On the other hand, some services just weren't ready for HN traffic and failed to scale up. Which.... they wouldn't, were they running on a cloud provider like AWS.

It's possible for any service to be surprisingly bursty, but I suspect more than you think are bursty on a regular basis.

P.S. DO Droplets are Digital Ocean's response to AWS Lambda, and are still "cloud."

Re: We reduced our server costs by moving away from AWS

#338

Post by pwinnski »

jjav wrote: post #280
pwinnski wrote: post #265

Earlier quoted context omitted.

Right, which is why I can't imagine moving from AWS to self-hosting. Having to hire a full set of Ops people to manage what AWS is managing for me would hurt.

How are you running AWS without people? Our AWS DevOps team isn't any smaller than they'd be if they were running a handful of servers in a colocation facility.

It's my former company now, but the answer is: For the initial workload that we migrated from the datacenter to AWS EC2 instances, we still had the existing DevOps people, and while they said things were a little easier, they were still pretty busy.

For all new dev, which eventually including migrated some functionality from that initial workload, we used AWS-specific managed services like Lambda, Dynamo, and SNS/SQS, which required MUCH less effort from DevOps. Two of the existing DevOps people were able to handle everything the so-called serverless development required in addition to their previous normal work.

I mean, the point of managed services is that the vendor takes care of managing them, so you don't need to deal with upgrades, security patches, etc, so if you're still doing just as much DevOps work, it's likely you're not taking good advantage of managed services!

So we had some initial effort to automate and standardize CloudFormation configuration, and then developers were able to launch new code bringing in millions in revenue with very minimal oversight from DevOps.

The advantage there, though, was mostly from "serverless" and managed services, not just EC2 vs datacenter. We still had a DBA for the Postgres stuff, we just didn't need one for the new Dynamo stuff.

Re: We reduced our server costs by moving away from AWS

#339

Post by hazmazlaz »

pessimizer wrote: post #56
hazmazlaz wrote: post #16

There is no way this figure is accurate. The annual spend cited of $1,000,000 is purely hypothetical, as admitted here: "However, all this data and processes need to happen on a server and, of course, we used AWS for it. A few years of growth later, we’re handling over 70,000 pages per minute, storing around 560 million pages, and paying well over $1,000,000 per year. Or at least we would be paying that much if we st…

> There is no way this figure is accurate. You've got the FUD covered, but you also need to add at least some substance to your claim. How do you know this figure would not be accurate? Why is your (hypothetical, not offered) estimate better than the author's?

Just as a basic sanity check, they claim to store 560 million pages of pure HTML. Internet Archive estimates the median HTML weight of a page to be ~20kb, which I fudged to 30kb in order to make a liberal estimate. Multiplied by the number of pages that would amount to approximately 16.8TB of data stored in s3. According to the AWS pricing calculator that would cost between $300-400 per month, and that's in s3 standard tier pricing with no intelligent tiering set up for less often retrieved data. I didn't bother calculating the compute cost for their rendering because I don't really understand how they do it, and also because the storage estimate was so far off of their claim that it didn't seem necessary to bother in order to refute the claim. I appreciate your challenge for additional evidence, and this is what I have to offer in response. I won't pretend like I did an exhaustive scientific study, this is more back of the napkin type stuff FWIW, but here it is.

Re: We reduced our server costs by moving away from AWS

#340

Post by boltzmann-brain »

dijit wrote: post #335
arinlen wrote: post #334

Earlier quoted context omitted.

> (...) an Operations Engineer (...) An operations engineer and a sysadmin are two entirely different roles. A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.

You're conflating an IT Technician and a Sysadmin. Sysadmins automate their work. Operations Engineer is just another title for sysadmin. Systems Engineer is also another title for the same thing. Platform Engineering (depending on where you are) is yet another variation. If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.

This whole string of conversation is some of the most horrendous navel gazing I've had in my comments for a while.
Post reply on HN