Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

101–110 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#101

Post by bhouston »

I wish this was a good condemnation of microservices in a general use case but it is very specific to the task at hand.

Honestly, the original architecture was insane though. They needed to monitor encoding quality for video streams so they decided to save each encoded video frame as a separate image on S3 and pass it around to various machines for processing.

That is a massive data explosion and very inefficient. It makes a lot more sense that they now look for defects directly on the machines that are encoding the video.

Another architecture that would work is to stream the encoded video from the encoding machines to other machines to decode and inspect. That would work as well. And again avoid the inefficiencies with saving and passing around individual images.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#102

Post by gondaloof »

rbanffy wrote: post #56
samwillis wrote: post #4

Next they will transition to on premises hardware from the cloud to save another 90%.... oh wait...

From Amazon's PoV, AWS is on-prem ;-)

Tongue in cheek, but

> Amazon Web Services, Inc. is a subsidiary of Amazon

So it’s technically another company.

Another comment seems to confirm this akshually comment ^_^’

https://news.ycombinator.com/item?id=35812230

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#103

Post by NBJack »

mjr00 wrote: post #62
wg0 wrote: post #20

Earlier quoted context omitted.

OR each team can write their features as a python package, rust crate, go module to be included in the main app. Libraries, versioned.

And then team 1 needs to upgrade pandas to 2.0, but team 2 is still on pandas 1, so when the main app pulls them in nothing works, so you need to start a cross-team committee to schedule the work to upgrade a single library... Separate services aren't a silver bullet, but as an fyi to the younger software developers, we tried "just have all teams work on the same code base and deployable artifact" for a long while an…

That's easy: we'll make the ultimate build system! It will scale, and maintain packages, and compile all the things transitively. Just give me $xx million dollars and a few years, and I'll give you the perfect solution.

Just press this button to start the upgrade build and....boom! 10,000 services and their dependencies being built on a ton of hardware; we can practically gurantee your change in dependency will be checked... Whoops, turns out your one dependency change cascaded into about 1.5% breakage....no, I don't know who owns those packsges; why do you ask? That's not my job!

/s

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#104

Post by Garlef »

ldargin wrote: post #65
LVB wrote: post #37

> Moving the solution to Amazon EC2 and Amazon ECS also allowed us to use the Amazon EC2 compute saving plans that will help drive costs down even further. So various parts of Amazon have to work through the AWS same pricing programs that the rest of us do?

Yes, to keep track of costs.

Also they might be actually be different legal entities.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#105

Post by debdut »

iamflimflam1 wrote: post #50

This really is a click bait title. They are talking about their video quality monitoring service, not their video streaming service. It’s something they use to check for defects in the video stream - hence the storing of individual frames in S3. Original title: Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%

The subtitle is "The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs." And the article itself mentions the 90% cost reduction. So the title seems pretty much in-line with the original intent.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#106

Post by shri_krishna »

The only time it makes sense to use edge/serverless anything is lightweight APIs and rendering HTML to end users so they get the page loaded as quickly as possible. That's the only use case good for edge. And any supporting infra that can help deliver rendered pages asap (like kv store on the edge for storing sessions, lightweight database on the edge for user profile data, queues etc). Anything that requires decent amount of processing should not live on the edge/serverless. It defeats the purpose.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#107

Post by bhouston »

ikiris wrote: post #6

Sending video frames between services is expensive, also doing per state transition hosting on things doing state transitions multiple times per second in a single stream is also expensive... Like, did they even think about cost when designing this the first time?

Yeah completely insane original design. A design I would expect from a first year intern who is just trying to make his first project work and is picking random technologies to string together.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#108

Post by simplotek »

bberrry wrote: post #8

I'd be surprised if this doesn't get taken down as it casts AWS lambda in an unfavorable light (and rightly so). That's the impression I have of Amazon's leadership but maybe I'm wrong.

> I'd be surprised if this doesn't get taken down as it casts AWS lambda in an unfavorable light (and rightly so).

The article mostly lays the blame on step functions. Also, lambdas are portrayed as event handlers that don't run relatively often. This means long running tasks that are ran occasionally, or events that don't fire that often. Once throughout needs go up or your invocation frequency comes closer to the millisecond then the rule of thumb is that you are already requiring a dedicated service.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#109

Post by Garlef »

quickthrower2 wrote: post #49
selcuka wrote: post #31

Earlier quoted context omitted.

It's the same story as NoSQL. "Let's migrate our transactional data that requires strict referential integrity to CouchDB... Oh, wait..."

Waiting for the next wave of tech misuse due to LLMs and ML!

There's a lot of room for "chinese-room" automation:

Write a verbal description of your cloud function and let the LLM simulate the execution.

Very cheap to develop. Very expensive to execute.

Post reply on HN