We reduced the AWS costs of our streaming data pipeline • Hacker News Board

Re: We reduced the AWS costs of our streaming data pipeline

#11

Post by acdha » Thu, May 28, 2020, 3:05 AM UTC

nojito wrote: post #10

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.

Re: We reduced the AWS costs of our streaming data pipeline

#12

Post by zaphirplane » Thu, May 28, 2020, 3:10 AM UTC

billyhoffman wrote: post #8

brianwawok wrote: post #7

Earlier quoted context omitted.

More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.

And doing this for months. Without noticing. Honestly this isn’t ultimately engineerings fault. This is a SaaS business Someone in their company is responsible for the COGS KPI. For that person to either not notice an increase in COGS, or to not be aggressively incentivizing engineering to reduce COGS, is giant red flag.

Ouch this thread ruined some poor souls promotion / pay rise narrative

Re: We reduced the AWS costs of our streaming data pipeline

#13

Post by QuinnyPig » Thu, May 28, 2020, 3:27 AM UTC

nojito wrote: post #10

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.

Re: We reduced the AWS costs of our streaming data pipeline

#14

Post by rlayton2 » Thu, May 28, 2020, 3:31 AM UTC

brianwawok wrote: post #7

dannyw wrote: post #6

Earlier quoted context omitted.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.

Perhaps more like ordering 100 lunches for your 100 employees and forgetting that 20 of them moved to night shift?

Re: We reduced the AWS costs of our streaming data pipeline

#15

Post by aritraghosh007 » Thu, May 28, 2020, 3:50 AM UTC

Back when AWS started, there would be articles about the work to master scalability and performance for the modern web but as things matured, we somehow ended up in a much larger heap of literature around AWS cost optimization.

Re: We reduced the AWS costs of our streaming data pipeline

#16

Post by sankalp221 » Thu, May 28, 2020, 4:05 AM UTC

brianwawok wrote: post #7

dannyw wrote: post #6

Earlier quoted context omitted.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.

Fun fact, 30-40% of the food in our supply chain is wasted. So, we're actually not too far from that. Sure, the meals aren't directly wasted by the company and rather through the supply chain. But, the waste is still there and is surprisingly high IMO.

Source: https://www.usda.gov/foodwaste/faqs

Re: We reduced the AWS costs of our streaming data pipeline

#17

Post by derex » Thu, May 28, 2020, 4:07 AM UTC

aritraghosh007 wrote: post #15

Back when AWS started, there would be articles about the work to master scalability and performance for the modern web but as things matured, we somehow ended up in a much larger heap of literature around AWS cost optimization.

In some sense this is a good problem to have. With on-prem you used to have very limited resources to start with, so cost efficiency is a baked-in requirement. With cloud providers you seem to have limitless resources and the new problem of cost optimization arises.

Admittedly there’s difference between optimizing fully-controlled resources and cloud provider managed services. For one, low visibility into cloud service internals makes such optimization harder.

Re: We reduced the AWS costs of our streaming data pipeline

#18

Post by vageli » Thu, May 28, 2020, 4:13 AM UTC

nojito wrote: post #10

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

> False equivalence. The engineer will be doing more than just cloud work.

> This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

Re: We reduced the AWS costs of our streaming data pipeline

#19

Post by owenmarshall » Thu, May 28, 2020, 4:22 AM UTC

alharith wrote: post #9

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram). Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to r…

I vouched for this comment because it’s a valid point and I’m not sure why it was killed.

I happen to disagree strongly, though: lots of engineers in my experience undervalue the work of systems administrators and underestimate the effort needed to operationalize any technology.

Running your own is absolutely fine if you are willing to keep your stack small and invest time learning the tools you pick. But there are still horror stories of people thinking snapshots are backups, turning the wrong knobs and turning off fsync on their databases, ...

Re: We reduced the AWS costs of our streaming data pipeline

#20

Post by runT1ME » Thu, May 28, 2020, 4:39 AM UTC

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job?

The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.