Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

371–380 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#371

Post by tylerdurden91 »

I think what most people are missing here is that they used AWS Step Functions in the wrong place. Part of the blame here is that in over enthusiasm of trying to get more users, AWS doesn't properly educate customers when to use which service. Worse, for each use case AWS has about dozens of options making the choice incredibly hard.

In this case, they probably should have used Step Functions Express, which charges based on duration as opposed to number of transitions and they're looking for "on host orchestration" like orchestrate a bunch of things which usually are done in small time and are done over & over many times. Step functions is better when workflows are running longer, and exactly once semantics are needed. Link for reading differences between Express & standard step functions: https://docs.aws.amazon.com/en_us/step-functions/latest/dg/c....

This also exemplifies the fact that I learned while being at Amazon & AWS that Amazon themselves dont know how best to use AWS. This being one of the great examples. I'll share 1 more:

- In my team within AWS, we were building a new service, and someone proposed to build a whole new micro service to monitor the progress of requests to ensure we dont drop requests. As soon I mentioned about visibility timeout in SQS queues, the whole need for the service went away. Saving Amazon money ($$) & time (also $$). But if I or someone else didn't mention, we would have built it.

I dont think serverless is a silver bullet, but I don't think this is a great example of when not to use serverless. It helps to know the differences between various services and when to use what.

PS: Ex Amazon & AWS here. I have nothing to gain or lose by AWS usage going up or down. I'm currently using a serverless architecture for my new startup which may bias my opinions here.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#372

Post by gtirloni »

NohatCoder wrote: post #358
gtirloni wrote: post #341

Earlier quoted context omitted.

Most companies will pay way more for the engineers maintaining their on-premises infrastructure than they would for AWS. On-premises still makes sense when you reach a certain scale. When you reach a certain scale.

They kind of need to be there anyway, physically maintaining servers turns out to be a miniscule part of the whole maintenance. If you really care about uptime you still need people on-call who can intervene as necessary.

It's not a minuscule part of a small company. I made the point that on-premises makes sense after a certain scale.

Once you have on-premises you need people that know switches, routers, rackmount server, hardware, virtualization, etc, plus keeping all of that properly maintained (security patches, IaC, periodic updates, analyzing performance, making sure it's properly architected, etc).

I often see people saying it's the same cost or less but it's really not. Unless you have no idea what you should be doing.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#374

Post by tylerdurden91 »

tyingq wrote: post #373

Seems somewhat curious that they didn't at least include Fargate. Feels like they jumped all the way from the typical overengineered setup into using AWS in a way that's very close to just "I need virtual machines".

Absolutely. Neither fargate nor step functions express. Seems like they did not evaluate all the options before making the jump.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#375

Post by tylerdurden91 »

LASR wrote: post #274

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed". Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. My candid reaction: Seriously? WTF? I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and…

To the contrary, from my time at Amazon, I felt that developers want to use more high level AWS services. Unfortunately, the landscape of AWS services is so rapidly evolving that Amazon engineers themselves cant keep up and end up using the wrong service.

As mentioned in other comments, there are options such as Fargate, that would still technically be "serverless" and still yield similar cost reductions. Not to mention that AWS also has Step functions express for "on host orchestration" use cases. This seems like a case where the original architecture wasn't very well researched and nor was the new one.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#376

Post by zamnos »

kypro wrote: post #282
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

It depends on your timing. If you're extremely unlucky, you'll buy the new set of servers and the recession will hit right after you sign the PO. Probability says you're not likely gonna be that unlucky, so the recession will hit probably elsewhere in the physical servers' life cycle. A recession hits, and now there's a focus on cutting costs. With AWS, you don't have much choice - if you stop paying the bill, the servers evaporate into the cloud. Physical servers don't. You can change their replacement schedule and just wait arm few years more to replace them. Hopefully the recession has passed by then and you can buy a whole new pile of servers.

Really though, it seems like a hybrid on-prem/cloud approach is one to consider. Software like Anthos eases this, though there are also pitfalls with this approach too.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#377

Post by The_Colonel »

dtech wrote: post #256
The_Colonel wrote: post #130

Earlier quoted context omitted.

I would frame it as a necessity rather than a benefit. Having siloed teams (services) is usually a problem which is better to avoid as much as you can.

Having autonomous teams is great for scaling and allowing everyone to go fast, without teams constantly blocking each other. Having hundreds of engineers work in a single monolith in a single repo without any kind of (enforced) boundaries is a one way ticket to a big ball of mud. You need to invest heavily in tooling to make it work, and e.g. Google does so. Having a network in between teams is a relatively easy way…

It allows everyone to go fast as long as the work is constrained within one service. It goes very slow once service / team coordination needs to happen and one team alone is not able to deliver the feature. This then often leads to services duplicating logic, amassing responsibilities in order to do as much as possible within "my" service to avoid this coordination bottleneck.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#378

Post by re-thc »

BbzzbB wrote: post #194
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

> Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

And then charging them to use AWS anywhere and outpost!

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#379

Post by tylerdurden91 »

tylerdurden91 wrote: post #371

I think what most people are missing here is that they used AWS Step Functions in the wrong place. Part of the blame here is that in over enthusiasm of trying to get more users, AWS doesn't properly educate customers when to use which service. Worse, for each use case AWS has about dozens of options making the choice incredibly hard. In this case, they probably should have used Step Functions Express, which charges b…

Worth mentioning as mentioned in other comments that moving video data around at that scale was a bad choice to begin with. They could have considered fargate and avoided moving the data around so much as well and realized similar reductions in cost. So the wins are not really coming from moving to monolith as much as they're coming from optimizing unnecessary data transfers.

If the article said fargate, which is technically still serverless we could have avoided a whole microservice vs monolith debate or serverless vs hosts/instances debate.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#380

Post by steveBK123 »

abluecloud wrote: post #309
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

It's honestly like a cult and a desire to want to "do it right" on AWS. The last few projects I've spent so much time setting up code deploy, load balancers, certificates, SES, route 53... This newest project, I've gone to heroku with everything being basically a few clicks to get setup.

So guys we need Lambdas + Step Functions + SES + SQS + SNS + MSK + AWS Batch + S3 + Lakeformation + Cloudformation + Athena + EMR + Redshift + Aurora + SageMaker + Cloudtrail + Codepipeline + maybe some EC2s to run AWS CLI on them.

Don't forget to configure Route53, VPC, IAM and an ELB.

Great - ready to start writing your app now?

Oh wow one of those components as configured with the other components isn't behaving as expected - time to contact AWS support!

Cynically I think CTOs see all this stuff and think they'll turn all their expensive on-shore devs into cheaper DevOps because AWS is magic and you don't need to write hard app code anymore.

I'd counter that AWS forces expensive on-shore devs into having to wear an entire new hat and be half a DevOps engineer to figure out how to make their code work on this alphabet soup instead of a Linux server.

Post reply on HN