Live data from Hacker News

We reduced the AWS costs of our streaming data pipeline

taloflow.ai

21–30 of 85 posts

Re: We reduced the AWS costs of our streaming data pipeline

#21

Back when AWS started, there would be articles about the work to master scalability and performance for the modern web but as things matured, we somehow ended up in a much larger heap of literature around AWS cost optimization.

We ended up here because Amazon can't scale. It's just uncool to admit you have to notice the pink elephant. Why? I don't know. Maybe it has to do with cred in engineering teams or for engineering teams in the broader org structure.

But the problem with AWS, with a lot of the "cloud", is the pitch that remote centralization of a service scales ad infinitum. It's still subject to the same constraints as self-managed, even if those constraints appear at a higher limit.

The greatest constraint is the per-unit pricing. You buy self-managed, you have huge upfront and period costs, but with remote, you see the $.03/MB price and assume that variable cost is more manageable over the long run. And it is... until price changes, overhead changes, bandwidth changes, or worse, accessibility changes. And suddenly, what you had cost-effective scaling on 18 months ago now has a massive deficit affixed to it. Because that's how most people used the platform... or because removing A or B features reduced maintenance costs or freed up bandwidth.

AWS is an experiment. Does it work in many or even most use cases? Yes. For now.

I love engineers. A lot. In fact, being in sales, I would give up a deal with an engineering team unless I knew for sure my ROI basis was solid. That said, I do know sales and marketing rhetoric. And having spent hundreds of hours in meetings with product, marketing and dev professionals, I wish I could record the stress-induced breakdowns I've seen in engineers and executives who had everything running buttery, "and then [provider] pushed [update]..." and they then have executives breathing on the back of their neck 16 hours a day, entire teams offline or unable to do basic tasks, etc. I just want to play that shit to people and say, "This is why you don't overpromise."

Re: We reduced the AWS costs of our streaming data pipeline

#22
I find it highly entertaining a two-year old company who was founded on the basis of helping slash cloud spending found so much waste in their own AWS spend. This is not an example of dogfooding, but an example of sheer incompetency and massive technical debt.

I'd really like to start seeing a series of blog posts from companies who are running extremely lean and efficient tech environments by utilizing cloud in an intelligent manner and avoiding the expensive and unnecessary bullshit that's so prevalent today. The ones that can brag "How we run a $4M/yr SaaS on $40k/yr of AWS spend!" are far more interesting than "How we stopped incinerating millions of VC money by simply turning off shit we didn't need"

Re: We reduced the AWS costs of our streaming data pipeline

#23
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.

That depends on the scale and other particulars that are only shared/known inside the company.

In my experience, beyond a certain scale it simply doesn't make any sense to use managed services anymore. There is a high initial upfront cost in development hours and hardware that is amortized over a very long time after, and this upfront cost is partially paid for by the reduced cloud bill.

Re: We reduced the AWS costs of our streaming data pipeline

#24
post #9

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram). Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to r…

Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry (granted you know how to use Docker).

Most small startups are actually the ones who don’t really need SaaS services.

Re: We reduced the AWS costs of our streaming data pipeline

#25
post #10

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

Yes. I’ve found that the amount you have to learn to use a managed service often equals or exceeds the amount you have to learn to run something on EC2 or on-prem. The automation/management costs of AWS or equivalent are a lot higher than people think and not significantly different from the costs to learn Linux and enough networking to do an “old-fashioned” deploy.

Re: We reduced the AWS costs of our streaming data pipeline

#26
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.

Most ETL processes should be declarative and configured in text files like you configure a CloudFormation or Terraform. Once you have that, the execution piece is relatively straightforward. I think a lot of issues and costs with ETL come from poor architecture decisions. For instance, we all made the mistake of running Extract processes with too many transformations. An extract should be an extract, transformation should come later.

Re: We reduced the AWS costs of our streaming data pipeline

#27

Earlier quoted context omitted.

More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.

And doing this for months. Without noticing. Honestly this isn’t ultimately engineerings fault. This is a SaaS business Someone in their company is responsible for the COGS KPI. For that person to either not notice an increase in COGS, or to not be aggressively incentivizing engineering to reduce COGS, is giant red flag.

The article did say that the motivation for doing so was because AWS credits were running out .. why prematurely optimize a free resource? :)

Re: We reduced the AWS costs of our streaming data pipeline

#28
post #6
post #5

Earlier quoted context omitted.

Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

> The whole point of AWS is to use services on demand

That's a decade-old misconception about how people actually use AWS.

Most servers I've seen in AWS are permanent.

In fact, it's an anti-pattern to wait until you need more capacity to scale up, since those servers may not be available, especially in newer instance families.

Even if the needed instances available, typically ASGs don't react the way you expect without a lot of experimentation (ie. outages.) An example is if traffic increases load, your health check may consider the servers to be unhealthy and start killing them, creating a death spiral.

Re: We reduced the AWS costs of our streaming data pipeline

#29
post #18
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

what is involved in managing backups? isn't that just a cronjob?
Post reply on HN