Live data from Hacker News

We reduced the AWS costs of our streaming data pipeline

taloflow.ai

41–50 of 85 posts

Re: We reduced the AWS costs of our streaming data pipeline

#41

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

This is probably a tradeoff in a lot of AWS related stuff; you pay a premium for convenience. But depending on your workload it can pay itself off fairly quickly; AWS bills can go up quite fast, whereas personnel costs are fairly predictable.

Re: We reduced the AWS costs of our streaming data pipeline

#42

It seems reasonable to make some of these cost comparisons more visible. ie If working on a new product or feature to understand upfront "this managed service is x% more then more bare bones" etc. essentially turning an alchemy into a science

AWS offers a cost calculator for just that purpose; they offer 'easier' products if you can't be arsed to dive into AWS costs and technologies yourself.

I think a lot of people make the mistake of assuming AWS is just an easy off-the-shelf thing you can just grab, but if you use it seriously it's a full-time job and its own expertise.

Source: I've done some AWS certifications, never was able to put them into practice though. I've also worked in multiple organizations that migrated to AWS, they all had a full-time team of people managing it.

It's a full-time, specialist job and you can't just palm it off to your engineers as a background thing.

Re: We reduced the AWS costs of our streaming data pipeline

#43
I am curious about the actual cost in $! Managing your own kafka or observability infra is expensive, you need a team to do this.

A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this.

Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.

Re: We reduced the AWS costs of our streaming data pipeline

#44

I am curious about the actual cost in $! Managing your own kafka or observability infra is expensive, you need a team to do this. A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this. Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.

Or course it is.

Where did we get the idea that engineers are hired to do only one thing?

This has never ever been the case in my experience.

Also Kafka being hard this manage is not the case. A simple look into many small companies and startups running their own clusters shows otherwise.

Re: We reduced the AWS costs of our streaming data pipeline

#45
post #18
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

Backups are a solved problem and has been for decades.

Can you think of a better example?

Re: We reduced the AWS costs of our streaming data pipeline

#46
post #11
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.

It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.

Re: We reduced the AWS costs of our streaming data pipeline

#47
post #44

I am curious about the actual cost in $! Managing your own kafka or observability infra is expensive, you need a team to do this. A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this. Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.

Or course it is. Where did we get the idea that engineers are hired to do only one thing? This has never ever been the case in my experience. Also Kafka being hard this manage is not the case. A simple look into many small companies and startups running their own clusters shows otherwise.

Engineers are hired to deliver and produce value. Tooling can be a part of it but if you can outsource something which is not your source of income, you have to do it. Engineering time is more valuable.

I also know many startups and small companies investing 5 people and 6 months to get an observability platform up and running while they could just get datadog or new relic for half the price... and I don't get into account outages and updates to the platform.

I remember a recent uber blog post on how they moved from build tool A to build tool B and a couple of weeks later, 3000 people where laid off. It's important to spend development time on revenue streams.

This is some nice piece of advice https://nav.al/build-a-team-that-ships

"Outsource everything that isn’t core. Resist the urge to pick up that last dollar. Founders do Customer Service."

Re: We reduced the AWS costs of our streaming data pipeline

#48
post #45
post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

Backups are a solved problem and has been for decades. Can you think of a better example?

They’re a solved problem in the sense that the tools exist but that still has an ongoing operational cost to correctly setup, secure, monitor and test – and there are plenty of examples of expensive failures or security breaches caused by people thinking it was an easy solved problem.

Deciding whether you get enough benefit from doing that yourself is a classic business trade off which any experienced engineer should consider.

Re: We reduced the AWS costs of our streaming data pipeline

#49
post #46
post #11

Earlier quoted context omitted.

It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.

It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.

That could be the answer to the question which was actually asked but if you read the thread again, notice that you’re arguing against a claim nobody made.

Re: We reduced the AWS costs of our streaming data pipeline

#50

Earlier quoted context omitted.

First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-…

You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand. Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

There are original-definition cookbooks and yet it still costs me time to provision my own lunch vs using the managed service of my corner restaurant.
Post reply on HN