Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the AWS costs of our streaming data pipeline

taloflow.ai

Post reply on HN
Print view View original

61–70 of 85 posts

Re: We reduced the AWS costs of our streaming data pipeline

#61

Post by brodouevencode »

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Agreed - there is a tradeoff that must factor in many things: engineer competency or the ability to get competent engineers, state of the product itself (maybe Elasticsearch as a service was an interim step in a longer term vision), complexity of the managed service itself, integratability (is that a word?) into other AWS services, maturity of the managed service, and probably a few other things I'm missing.

We've seen our teams go both from managed to non-managed and non-managed to managed with relative success - to give scale, across all of our accounts we spend way north of $3 million/month at AWS so this has happened within our realm a quite few times. The short, unsatisfying answer is that _it depends_. We have an internal policy from the suits that "if there's a managed version, use it" but most of our teams are thankfully smart enough to take that at face value and do their own analysis.

Re: We reduced the AWS costs of our streaming data pipeline

#62

Post by brodouevencode »

runT1ME wrote: post #20
QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job? The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.

Unfortunately that's not nearly how AWS works. AWS breaks down everything and charges you for it separately. Flink and Glue are entirely different animals.

Re: We reduced the AWS costs of our streaming data pipeline

#64

Post by brodouevencode »

nojito wrote: post #10
QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

While I don't entirely disagree I think it should be made clear that both can be true, even at the same time. To spin up a 9 node managed Elasticsearch cluster load-balanced across two regions takes a competent engineer with practice roughly a couple of hours, or twenty to thirty minutes if they were smart and terraformed it out previously. Now there's a whole host of potential problems that come along with using that managed Elasticsearch cluster too (no access to "cluster mode", no tunability, etc). But if those potential problems don't apply to you and a very vanilla ES cluster suits your use case then you're fine.

Alternatively that practiced engineer could have spun up a self-managed ES cluster in a couple of DCs in about the same time, but now has the obligation to maintain those servers (patching, etc.). Maybe that marginal cost is damn near zero - chef has been deployed to all instances and enforces patching and there's already good security monitoring in place, etc. The cost of that engineer managing that box, as with a managed ES in AWS, is practically nothing.

TL;DR: as in all cases, it depends.

Re: We reduced the AWS costs of our streaming data pipeline

#65

Post by strgcmc »

Neil44 wrote: post #40
vageli wrote: post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

How much liability can you claim against AWS if there's an issue with their RDS backups?

Liability? Probably none. See section "11. Limitations of Liability." here: https://aws.amazon.com/agreement/

RDS SLA's are here (doesn't mention backups though, so not sure how that's handled): https://aws.amazon.com/rds/sla/

Not a lawyer or anything, but my layman's understanding is that, you essentially are voluntarily opting-in to waiving liability when you sign up for AWS and accept the terms and conditions, and instead of liability, you agree to accept service credits if SLAs are not met.

Re: We reduced the AWS costs of our streaming data pipeline

#66

Post by OldHand2018 »

user5994461 wrote: post #38
dylan604 wrote: post #29

Earlier quoted context omitted.

what is involved in managing backups? isn't that just a cronjob?

Need a lot of storage and make sure the backup is readable (view the files content or try a restore). The number one backup solutions nowadays is AWS S3, because it's easy-to-use unlimited storage. How does a company handle backups without S3? Usually they don't. That would require employees to buy machines/SAN with tens of TB of storage and maintain them (weeks in ordering and travelling to the datacenter once in a…

Easy to use unlimited storage is a sure recipe for not finding what you actually need, restoring the wrong backup, etc.

Unless you take your DR plans seriously, the cloud doesn't eliminate risk, it just changes it.

The place I work at forces a failover on a monthly basis, and does a full-on offsite DR exercise twice a year.

I'm sure it took time to set it all up, but now that it's there it takes almost no effort to continue.

Re: We reduced the AWS costs of our streaming data pipeline

#67

Post by sushshshsh »

serverQuestion wrote: post #55
Neil44 wrote: post #40

Earlier quoted context omitted.

How much liability can you claim against AWS if there's an issue with their RDS backups?

How much liability can you claim against Cloud Employee if there's an issue with your RDS backups?

What is the chance that Amazon cause an issue with your RDS backup, versus a Cloud Employee?

The answer is definitely not clear to me at all

EDIT: no sarcasm, I legitimately don't know which I would choose as a biz owner

Re: We reduced the AWS costs of our streaming data pipeline

#68

Post by ec109685 »

sokoloff wrote: post #50
ec109685 wrote: post #39

Earlier quoted context omitted.

You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand. Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

There are original-definition cookbooks and yet it still costs me time to provision my own lunch vs using the managed service of my corner restaurant.

Yes, managed services are better in many cases.

Re: We reduced the AWS costs of our streaming data pipeline

#69

Post by gberger »

brodouevencode wrote: post #61
QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Agreed - there is a tradeoff that must factor in many things: engineer competency or the ability to get competent engineers, state of the product itself (maybe Elasticsearch as a service was an interim step in a longer term vision), complexity of the managed service itself, integratability (is that a word?) into other AWS services, maturity of the managed service, and probably a few other things I'm missing. We've se…

> integratability

interoperability?

Re: We reduced the AWS costs of our streaming data pipeline

#70

Post by [deleted] »

sokoloff wrote: post #50
ec109685 wrote: post #39

Earlier quoted context omitted.

You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand. Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

There are original-definition cookbooks and yet it still costs me time to provision my own lunch vs using the managed service of my corner restaurant.

[deleted]
Post reply on HN