Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

271–280 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#271

Post by moonchrome »

ojkelly wrote: post #254
debdut wrote: post #105

Earlier quoted context omitted.

The subtitle is "The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs." And the article itself mentions the 90% cost reduction. So the title seems pretty much in-line with the original intent.

But, by omission is reads that Prime Video rebuilt their stack without serverless and got a 90% cost reduction. This post is going to pick up a lot of traction and I suspect these comments are going to bikeshed monolith vs microservices for the next day. On reading it, this is for a video quality monitoring system, that needs to consume and process video. Generally a compute and time intensive task. Something not alw…

Serverless has such bullshit insidious pricing that makes it seem like you're saving money only to figure out you're in shit once you're knee deep in it.

For example you'll have to read fine print to find out that 256MB lambda will have the compute power of a 90s desktop PC because compute scales with memory. And to get access to "one core" of compute you have to use like 2GB of memory.

Now you may say "serverless isn't geared towards compute" - but this kind of CPU bottlenecking affects rudimentary stuff - like using any framework that does some upfront optimizations will murder your first request/cold start performance - EF Core ORM expression compiler will take seconds to cold start the model/queries ! For comparison I can run ~100 integration tests (with entire context bootstrap for each) against a real database in that time on my desktop machine. It's unbelievably slow - unless you're doing trivial "reparse this JSON and manually concat shit to a DB query" kind of workloads.

You could say those frameworks aren't suited for serverless - or you could say that the pricing is designed to screw over people trying to port these kinds of workloads to serverless.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#272

Post by nodefortytwo »

BbzzbB wrote: post #194
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

To be fair to AWS, they do work really hard to (at least at an account level) to optimize workloads with you. They do this so overall you'll move more workloads to them.

its quite simple, if workload x can be done 100% cheaper on-prem then its an obvious move (probably) if AWS manage to get that closer to 30-40% then the operational benefits of using AWS make more sense, more workloads, more total spend.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#273

Post by GabeIsko »

jon-wood wrote: post #224
Aeolun wrote: post #131

Earlier quoted context omitted.

I feel like it’s an object lesson in using the right solution for a problem. Step functions do not appear to me to be something that you’d use for things that need to be executed multiple times per second.

Having occasionally looked at them for workflow driven tasks I'm not sure what the use case for Step Functions is, unless your workflow being called once an hour or something they seem infeasibly expensive for what they offer, and somehow manage to be more complex than just writing some code to model the workflow.

It's a BPM product. If you have a highly regulated workflow that has to be changed a lot by multiple parties, these products start to make sense. The AWS step functions aren't that great in the BPM and workflow automation either, but I imagine AWS just wants to have a first party offering.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#274

Post by LASR »

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed".

Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory.

My candid reaction: Seriously? WTF?

I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and then back down to run some buffer computations. Fixing the problem and then calling it a win?

But I think I understand what might have lead to this. At AWS, there is an emphasis on using their own services. So when use cases that don't fit well on top of AWS services come up, there is internal pressure to shoehorn it anyway. Hence these sorts of decisions.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#275

Post by withinboredom »

anonzzzies wrote: post #269
withinboredom wrote: post #180

Earlier quoted context omitted.

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.

There are some extreme circumstances where pushing broken tests to production make sense. For example, if you push a simple change to simply `return false` and disable a feature in code. In this case, the tests using it will probably fail but the desired behavior happens in production. At this point, you have a bit more time to set the tests to 'skip' while the load is shed in production. Even if you break tests on purpose, you should fix them asap as you are blocking literally every other team in the company. Thus, you need a really good reason to do so (like if you didn't do it asap, global downtime would ensue).

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#276

Post by exitheone »

__turbobrew__ wrote: post #140
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

It's not only tests, a rollback because your pdf generation has a bug will also mean rolling back for example a customer facing new API, slowing down the API team until the monolith is fixed or the change reverted and rebuilt.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#277

Post by chank »

LASR wrote: post #274

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed". Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. My candid reaction: Seriously? WTF? I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and…

> Fixing the problem and then calling it a win?

It is a win. Just not the win they're aluding to.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#278

Post by tough »

sverhagen wrote: post #243
Paul-Craft wrote: post #214

Earlier quoted context omitted.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

If you talk about no vendor lock in, and you'd want to take your database then to a competitor, like Google Cloud, Azure, or on-premises, wouldn't you exactly expect to do a SQL dump and reloading everything? To me, the one-click move from RDS to Aurora you describe is a nice shortcut, but it doesn't invalidate that you can still do the former if you wanted to move to the competitor. Vendor lock in seems more that yo…

Seems a bit of a dark pattern to have shortcut to onboard and not offer the same shortcut to offboard.

Just like easy subscribe-online publications that will have you call during 2 hours with a rep pushing you discounts or whatever to cancel such subscription.

Just not cool

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#279

Post by kristopolous »

freeone3000 wrote: post #61
kristopolous wrote: post #55

Earlier quoted context omitted.

There's a bunch of things I'd like to have them do. If they could span across machines like clusters that would be amazing. If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing. If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool. If I could…

Your asks seem easily answered with docker + kubernetes. Actually, this is in fact the use case for kubernetes — a fault tolerant distributed system running arbitrarily, simply packaged code. This has to be what you’ve tried — what issue are you running into?

trying to using the actual software to accomplish these actual tasks. You're right though, that is the promise of the software - it doesn't deliver.

I wish I had infinite time to document all the issues. This isn't a small nuanced detailed thing - it falls deeply, systemically fundamentally short and in practice you still get the magical monolithic system it tried to kill but now with more obfuscation, complexity and a theatrical slight of hand to convince yourself it isn't that.

Instead of the server being configured for the monolithic app, it's now extensively and carefully configured for the myriad of containers, hostnames, configurations and connections of the containers running the microservice app.

It's in practice the same problem with a different costume.

The other promise of it being a collection of smaller constrained services running on tcp ports talking to each other ... that's nothing new. You've invented the idea of computer networks.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#280

Post by datadeft »

bootsmann wrote: post #241
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)

It is possible[1]. You are better off this way than the AWS "native" way.

1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...

Post reply on HN