Live data from Hacker News

We reduced the AWS costs of our streaming data pipeline

taloflow.ai

31–40 of 85 posts

Re: We reduced the AWS costs of our streaming data pipeline

#31
post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

what is involved in managing backups? isn't that just a cronjob?

There was a time when I used to think the same, then I found my backups were corrupted (or stopped because run out of space, etc) just in the moment when I needed them.

Re: We reduced the AWS costs of our streaming data pipeline

#33
post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

what is involved in managing backups? isn't that just a cronjob?

First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-replica design, where the node that runs the cron job may vary based on the current topology?

And if you need some sort of cluster-aware lock to coordinate backups among different peers, you'll need to decide which system works for you, implement it, and maintain that as a separate system. And if that needs to be upgraded, figure out a bulletproof process for upgrading it while it's still being used as a coordinator.

Then, you need to ensure there's storage for the backup. You need to decide what kind of storage you're going to use, make sure you've got enough space, figure out how to encrypt the storage (very important in secure environments), how to protect the storage using authn/authz. And lots of environments have retention and storage lifecycle policies - you don't want to put the old backups on the expensive fast media; you want it on the cheap slow media. And some environments make you dispose of old data, so you have to figure out how to age it out but without ever losing the backups you want to keep.

Finally, you need to make sure the backups you create are valid and usable. So you'll want to build an automated regression testing procedure to ensure that every time you make a change (regardless of how minor) to the system being backed up or the backup process, that you end up with usable backups.

(Disclaimer: I work for AWS, but opinions expressed here are my own and not necessarily those of my employer.)

Re: We reduced the AWS costs of our streaming data pipeline

#34
post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

what is involved in managing backups? isn't that just a cronjob?

Not for a reliable solution. For example, assume you have a master and a replica database for reliability, what happens if the master, where the cron runs, fails? Do you remember to set up the cron up on the replica? From my experience, having worked on backup software, over 10% of servers that need to be backed up are not.

System reliability is hard, and the cloud makes that easier.

Re: We reduced the AWS costs of our streaming data pipeline

#35

Earlier quoted context omitted.

what is involved in managing backups? isn't that just a cronjob?

First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-…

“managed services” don’t necessarily save you the headache of making sure backups are usable. But other points are valid.

Re: We reduced the AWS costs of our streaming data pipeline

#36
post #6

Earlier quoted context omitted.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

> The whole point of AWS is to use services on demand That's a decade-old misconception about how people actually use AWS. Most servers I've seen in AWS are permanent. In fact, it's an anti-pattern to wait until you need more capacity to scale up, since those servers may not be available, especially in newer instance families. Even if the needed instances available, typically ASGs don't react the way you expect witho…

This is why service health endpoints should be very carefully designed, with the proper balance between health inspection and performance penalty.

That being said, it's true that an ALB doesn't offer throttling capability like a true reverse proxy such as HAProxy provides, were you can cap the number of concurrent requests and give a chance to your backend to avoid death by overload.

I wish there would be a way for ASGs to at least make the distinction between an unhealthy instance and an overloaded one.

Re: We reduced the AWS costs of our streaming data pipeline

#37
post #22

I find it highly entertaining a two-year old company who was founded on the basis of helping slash cloud spending found so much waste in their own AWS spend. This is not an example of dogfooding, but an example of sheer incompetency and massive technical debt. I'd really like to start seeing a series of blog posts from companies who are running extremely lean and efficient tech environments by utilizing cloud in an i…

two-year old companies have limited resources. It might have been a deliberate trade off to focus on work that produces value to the customers.

Maybe the blog post would have been "How we run a $1M/yr SaaS on $40k/yr of AWS spend!" instead of $4M?

Re: We reduced the AWS costs of our streaming data pipeline

#38
post #18

Earlier quoted context omitted.

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

what is involved in managing backups? isn't that just a cronjob?

Need a lot of storage and make sure the backup is readable (view the files content or try a restore).

The number one backup solutions nowadays is AWS S3, because it's easy-to-use unlimited storage.

How does a company handle backups without S3? Usually they don't. That would require employees to buy machines/SAN with tens of TB of storage and maintain them (weeks in ordering and travelling to the datacenter once in a while). It's too much hassle so nevermind.

Re: We reduced the AWS costs of our streaming data pipeline

#39

Earlier quoted context omitted.

what is involved in managing backups? isn't that just a cronjob?

First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-…

You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand.

Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

Re: We reduced the AWS costs of our streaming data pipeline

#40
post #18
post #10

Earlier quoted context omitted.

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

How much liability can you claim against AWS if there's an issue with their RDS backups?
Post reply on HN