We reduced the AWS costs of our streaming data pipeline
1–10 of 85 posts
Re: We reduced the AWS costs of our streaming data pipeline
#2"Eliminate unused EC2 instances" -27% of cost
Haha, so cleaned the internal IT / DevOps mess and call it a day and than blog post it
Re: We reduced the AWS costs of our streaming data pipeline
#3Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?
Re: We reduced the AWS costs of our streaming data pipeline
#4We went through a similar process with GCP, which was annoying since GCP was sold as being cheaper than AWS.
Re: We reduced the AWS costs of our streaming data pipeline
#5"Eliminate unused EC2 instances" -27% of cost Haha, so cleaned the internal IT / DevOps mess and call it a day and than blog post it
Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.
Re: We reduced the AWS costs of our streaming data pipeline
#6"Eliminate unused EC2 instances" -27% of cost Haha, so cleaned the internal IT / DevOps mess and call it a day and than blog post it
Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.
I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess.
The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.
Re: We reduced the AWS costs of our streaming data pipeline
#7Earlier quoted context omitted.
Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.
I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.
More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.
Re: We reduced the AWS costs of our streaming data pipeline
#8Earlier quoted context omitted.
I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.
More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.
And doing this for months. Without noticing.
Honestly this isn’t ultimately engineerings fault. This is a SaaS business Someone in their company is responsible for the COGS KPI. For that person to either not notice an increase in COGS, or to not be aggressively incentivizing engineering to reduce COGS, is giant red flag.
Re: We reduced the AWS costs of our streaming data pipeline
#9Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?
Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram).
Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to run their own these days.
Re: We reduced the AWS costs of our streaming data pipeline
#10Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?
False equivalence. The engineer will be doing more than just cloud work.
This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money