Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the AWS costs of our streaming data pipeline

taloflow.ai

Post reply on HN
Print view View original

71–80 of 85 posts

Re: We reduced the AWS costs of our streaming data pipeline

#71

Post by otterley »

ec109685 wrote: post #39
otterley wrote: post #33

Earlier quoted context omitted.

First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-…

You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand. Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

Sometimes there are cookbooks, but they are of varying quality and often don't have dedicated resources to maintain them, so I would use them with great caution. You also have to implement them and often maintain the underlying infrastructure.

But I was really responding to the brusque naiveté of the "just write a cronjob" response.

Re: We reduced the AWS costs of our streaming data pipeline

#72

Post by Neil44 »

sushshshsh wrote: post #67
serverQuestion wrote: post #55

Earlier quoted context omitted.

How much liability can you claim against Cloud Employee if there's an issue with your RDS backups?

What is the chance that Amazon cause an issue with your RDS backup, versus a Cloud Employee? The answer is definitely not clear to me at all EDIT: no sarcasm, I legitimately don't know which I would choose as a biz owner

I would personally always have an off-platform backup to fall back on, as protection against the platform going down, accidental damage to data or malicious damage to data. Snapshots in cold storage too.

Re: We reduced the AWS costs of our streaming data pipeline

#73

Post by cloudfalcon »

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Hi Corey! I'm the author of the blog post - I definitely agree with you. 9 times out of 10 engineering teams underestimate cost of engineering work as well as opportunity cost lost due to managing non-core functionality internally or moving away from managed services.

For us our pipeline was actually easier to work with Flink than Glue because of the restrictions that Amazon placed on it and so that factored into our decision.

Re: We reduced the AWS costs of our streaming data pipeline

#74

Post by jbn »

owenmarshall wrote: post #54
pmelendez wrote: post #31

Earlier quoted context omitted.

There was a time when I used to think the same, then I found my backups were corrupted (or stopped because run out of space, etc) just in the moment when I needed them.

I once worked at an IT shop that worked closely with the construction industry. A new sports stadium was being built and we were doing panoramic photos during each stage of construction, and rendering them in a web app where the facilities team could “peel back the layers” and see what was behind the wall or under the floor all the way down to the foundation. This was... 15 years ago? So it was pretty neat technology…

One wonders why tar even has an option (and a default!) to not recurse. What would be the use case for that?

The "normal" use case seems to have a recursive archival. Sounds like somebody chose the wrong default... would be an interesting software archaelogy project to figure out where this "feature" originated.

Re: We reduced the AWS costs of our streaming data pipeline

#75

Post by csharptwdec19 »

alharith wrote: post #9
QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram). Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to r…

Sometimes.

But developers are part of this problem too. There's plenty of times where I see devs immediately reach for tools instead of learning just a little bit more about what they already have. My favorite example is when folks want to add a NoSQL db into the mix on top of a traditional db. Not because there's a real performance need, but because for their use case it is 'easier'. Never mind that their problem possibly could have been solved by just writing their own SQL instead of trusting a garbage ORM...

Re: We reduced the AWS costs of our streaming data pipeline

#76

Post by csharptwdec19 »

brodouevencode wrote: post #63
tyingq wrote: post #60

The initial pie chart seems to indicate that either AWS glue is significantly overpriced, or that they were doing something wrong.

As with all things AWS the more "magic" there is to it, the more expensive it is.

Huge part of why I always try to build applications as platform agnostic as possible.

If I make a .NET service or site, I know (with the tools I use) I can deploy it on any linux or windows machine without issue. I can take it anywhere that I can run any software.

Sure, may need more glue for certain scenarios, but you know that you can move as soon as a provider shows it's fangs.

Re: We reduced the AWS costs of our streaming data pipeline

#77

Post by user5994461 »

OldHand2018 wrote: post #66
user5994461 wrote: post #38

Earlier quoted context omitted.

Need a lot of storage and make sure the backup is readable (view the files content or try a restore). The number one backup solutions nowadays is AWS S3, because it's easy-to-use unlimited storage. How does a company handle backups without S3? Usually they don't. That would require employees to buy machines/SAN with tens of TB of storage and maintain them (weeks in ordering and travelling to the datacenter once in a…

Easy to use unlimited storage is a sure recipe for not finding what you actually need, restoring the wrong backup, etc. Unless you take your DR plans seriously, the cloud doesn't eliminate risk, it just changes it. The place I work at forces a failover on a monthly basis, and does a full-on offsite DR exercise twice a year. I'm sure it took time to set it all up, but now that it's there it takes almost no effort to c…

The cloud eliminates the most common risks, that is ops + developers simply giving up on backups because there is nowhere to store them, and not being able to access them anyway.

Typical new sysadmin in large corp: The backup storage is full and backups have been failing since before I joined, should we do something about it?

Oh we raised tickets to request more disks. They will take months to arrive if they ever pass approvals.

Re: We reduced the AWS costs of our streaming data pipeline

#78

Post by brodouevencode »

csharptwdec19 wrote: post #76
brodouevencode wrote: post #63

Earlier quoted context omitted.

As with all things AWS the more "magic" there is to it, the more expensive it is.

Huge part of why I always try to build applications as platform agnostic as possible. If I make a .NET service or site, I know (with the tools I use) I can deploy it on any linux or windows machine without issue. I can take it anywhere that I can run any software. Sure, may need more glue for certain scenarios, but you know that you can move as soon as a provider shows it's fangs.

Speaking from experience - not a bad idea.

Re: We reduced the AWS costs of our streaming data pipeline

#79

Post by brodouevencode »

gberger wrote: post #69
brodouevencode wrote: post #61

Earlier quoted context omitted.

Agreed - there is a tradeoff that must factor in many things: engineer competency or the ability to get competent engineers, state of the product itself (maybe Elasticsearch as a service was an interim step in a longer term vision), complexity of the managed service itself, integratability (is that a word?) into other AWS services, maturity of the managed service, and probably a few other things I'm missing. We've se…

> integratability interoperability?

Thanks :)

Re: We reduced the AWS costs of our streaming data pipeline

#80

Post by dan_quixote »

QuinnyPig wrote: post #3

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

My personal favorite is "Move off of AWS MSK". No big deal - we just fire up some kafka brokers and and zookeeper nodes in ECS! All we gotta do is run several more supporting services to keep the cluster healthy and deal with the nightmare of Apache security ourselves.

As far as I'm concerned MSK is cheap - one broker is priced roughly same price as 2 equivalent EC2 instances. And you don't have to worry about zookeeper at all!

Post reply on HN