Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?
We reduced the AWS costs of our streaming data pipeline
41–50 of 85 posts
Re: We reduced the AWS costs of our streaming data pipeline
#42It seems reasonable to make some of these cost comparisons more visible. ie If working on a new product or feature to understand upfront "this managed service is x% more then more bare bones" etc. essentially turning an alchemy into a science
I think a lot of people make the mistake of assuming AWS is just an easy off-the-shelf thing you can just grab, but if you use it seriously it's a full-time job and its own expertise.
Source: I've done some AWS certifications, never was able to put them into practice though. I've also worked in multiple organizations that migrated to AWS, they all had a full-time team of people managing it.
It's a full-time, specialist job and you can't just palm it off to your engineers as a background thing.
Re: We reduced the AWS costs of our streaming data pipeline
#43A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this.
Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.
Re: We reduced the AWS costs of our streaming data pipeline
#44I am curious about the actual cost in $! Managing your own kafka or observability infra is expensive, you need a team to do this. A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this. Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.
Where did we get the idea that engineers are hired to do only one thing?
This has never ever been the case in my experience.
Also Kafka being hard this manage is not the case. A simple look into many small companies and startups running their own clusters shows otherwise.
Re: We reduced the AWS costs of our streaming data pipeline
#45Earlier quoted context omitted.
False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money
> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.
Can you think of a better example?
Re: We reduced the AWS costs of our streaming data pipeline
#46Earlier quoted context omitted.
False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money
It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.
Re: We reduced the AWS costs of our streaming data pipeline
#47I am curious about the actual cost in $! Managing your own kafka or observability infra is expensive, you need a team to do this. A 67% reduction doesn't say the whole truth. They have more services to manage now, which means they need more people and more time to do this. Saving 10k from your AWS bill by hiring 2 more engineers is not cost effective.
Or course it is. Where did we get the idea that engineers are hired to do only one thing? This has never ever been the case in my experience. Also Kafka being hard this manage is not the case. A simple look into many small companies and startups running their own clusters shows otherwise.
I also know many startups and small companies investing 5 people and 6 months to get an observability platform up and running while they could just get datadog or new relic for half the price... and I don't get into account outages and updates to the platform.
I remember a recent uber blog post on how they moved from build tool A to build tool B and a couple of weeks later, 3000 people where laid off. It's important to spend development time on revenue streams.
This is some nice piece of advice https://nav.al/build-a-team-that-ships
"Outsource everything that isn’t core. Resist the urge to pick up that last dollar. Founders do Customer Service."
Re: We reduced the AWS costs of our streaming data pipeline
#48Earlier quoted context omitted.
> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.
Backups are a solved problem and has been for decades. Can you think of a better example?
Deciding whether you get enough benefit from doing that yourself is a classic business trade off which any experienced engineer should consider.
Re: We reduced the AWS costs of our streaming data pipeline
#49Earlier quoted context omitted.
It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.
It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.
Re: We reduced the AWS costs of our streaming data pipeline
#50Earlier quoted context omitted.
First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-…
You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand. Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.