We reduced the AWS costs of our streaming data pipeline • Hacker News Board

Re: We reduced the AWS costs of our streaming data pipeline

#81

Post by glogla » Thu, May 28, 2020, 9:43 PM UTC

redis_mlc wrote: post #28

dannyw wrote: post #6

Earlier quoted context omitted.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

> The whole point of AWS is to use services on demand That's a decade-old misconception about how people actually use AWS. Most servers I've seen in AWS are permanent. In fact, it's an anti-pattern to wait until you need more capacity to scale up, since those servers may not be available, especially in newer instance families. Even if the needed instances available, typically ASGs don't react the way you expect witho…

I think there's different levels of "on demand". In this case, I think this is much slower level of "on demand".

I see two primary use cases of cloud:

1) You're a startup or just need something small, and want to focus on building your MVP, instead of messing around with colocated Linux servers. Cloud is much more expensive than those, but you don't care because you don't really need all that much, or maybe you are VC backed and have unlimited money.

2) You're a large company with broken internal processes. You can get server in company datacenter in three months after seven approvals (since it's capex), or you can spin up an EC2 instance. You don't care about cost since you have unlimited money.

Those are kind of medium scale "on demand" - not "I need 100 new servers right this minute" but "I need server in ten minutes intead of 'when I get to buy one' or 'in three months and 37 forms'".

In both cases, you're throwing money away, because time is more important for you than the extra money cloud costs.

Re: We reduced the AWS costs of our streaming data pipeline

#82

Post by glogla » Thu, May 28, 2020, 9:59 PM UTC

sidlls wrote: post #23

QuinnyPig wrote: post #13

Earlier quoted context omitted.

I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.

That depends on the scale and other particulars that are only shared/known inside the company. In my experience, beyond a certain scale it simply doesn't make any sense to use managed services anymore. There is a high initial upfront cost in development hours and hardware that is amortized over a very long time after, and this upfront cost is partially paid for by the reduced cloud bill.

That also depends on the markup.

EKS on EC2 costs EC2 costs + flat money for control plane, so that might make more sense than running your own Kubernetes on EC2. (although I have no experience to say how much time this actually saves you)

Managed Kafka costing 2x the cost of EC2 infrastructure? Probably not worth it.

Re: We reduced the AWS costs of our streaming data pipeline

#83

Post by tjbiddle » Fri, May 29, 2020, 2:31 AM UTC

dannyw wrote: post #6

tjbiddle wrote: post #5

Earlier quoted context omitted.

Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

I don't disagree with it that at all. By all means, we should be doing our best not to waste resources.

Just stating that the waste was only 5% of the total savings.

Re: We reduced the AWS costs of our streaming data pipeline

#84

Post by runT1ME » Fri, May 29, 2020, 3:57 AM UTC

brodouevencode wrote: post #62

runT1ME wrote: post #20

Earlier quoted context omitted.

If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job? The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.

Unfortunately that's not nearly how AWS works. AWS breaks down everything and charges you for it separately. Flink and Glue are entirely different animals.

Can you explain more? From what I understand, if you're already running a Flink cluster on AWS and you have capacity for another job, you aren't charged more, no?

I haven't used Glue, but it seems like it's able to do stream processing on Kinesis and dumping to S3 or whatnot, so it seems like there's overlap with using EMR running Flink?

Re: We reduced the AWS costs of our streaming data pipeline

#85

Post by chrischen » Fri, May 29, 2020, 7:15 AM UTC

ckdarby wrote: post #57

chrischen wrote: post #24

Earlier quoted context omitted.

Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry (granted you know how to use Docker). Most small startups are actually the ones who don’t really need SaaS services.

>Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry This has not been the case at multiple employers and or consulting clients. If you're providing software to an enterprise this almost will never fly. That single docker container will have an outage when basically anything happens. The container dies, systemd fails to restart, node dies, network switch d…

I think you're misinterpreting my comment. I meant specifically for most small time startups, not necessarily small time startups deploying enterprise apps. If you're deploying enterprise apps then by definition you're for all intents and purposes "fb scale."

A lot of SaaS promise infinite scalability—a need which often never comes to most small time startups.