Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

61–70 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#61

Post by freeone3000 »

kristopolous wrote: post #55
wg0 wrote: post #14

Earlier quoted context omitted.

As for me,I have been trying to discover that 5% that cannot be done without microservices.

There's a bunch of things I'd like to have them do. If they could span across machines like clusters that would be amazing. If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing. If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool. If I could…

Your asks seem easily answered with docker + kubernetes. Actually, this is in fact the use case for kubernetes — a fault tolerant distributed system running arbitrarily, simply packaged code. This has to be what you’ve tried — what issue are you running into?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#62

Post by mjr00 »

wg0 wrote: post #20
alkibiades wrote: post #15

Earlier quoted context omitted.

microservices make a lot of sense organizationally where each feature team can own their own feature service.

OR each team can write their features as a python package, rust crate, go module to be included in the main app. Libraries, versioned.

And then team 1 needs to upgrade pandas to 2.0, but team 2 is still on pandas 1, so when the main app pulls them in nothing works, so you need to start a cross-team committee to schedule the work to upgrade a single library...

Separate services aren't a silver bullet, but as an fyi to the younger software developers, we tried "just have all teams work on the same code base and deployable artifact" for a long while and it didn't work very well either.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#63

Post by sgtnoodle »

ad-astra wrote: post #7

Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.

Indeed, it does seem rather ridiculous at face value. On the other hand, I have coworkers that run CPU-IPC bound workloads inside x86-64 docker containers on M1 macs (incurring the overhead of both machine code emulation and OS virtualization). I have other coworkers sweating for hours whether to use 32-bit or 64-bit integers for APIs designed for microcontrollers running at 300Mhz. I have even more coworkers writing stuff in rust because it's "memory safe" and "so fast", but they have no idea that they're doing thousands of unnecessary heap memory allocations per second when I naively start asking questions in a code review.

Even really smart, capable people in general have really poorly calibrated intuition when it comes to the intrinsic overhead of software. It's a testament to the raw computational power of modern hardware I guess. In the case of AWS, it's never been easier to accidentally a million dollars a month.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#64

Post by bcrosby95 »

avereveard wrote: post #40
wg0 wrote: post #20

Earlier quoted context omitted.

OR each team can write their features as a python package, rust crate, go module to be included in the main app. Libraries, versioned.

Yeah I see a lot of things that could be libraries packaged as services, so now each invocation incur in network latency and every transaction needs a two phase commit. And because each service need its own replica, deployment pipeline, and versioned internal api, production and deployment cost skyrocket

Because, you see, if you surround shit with other shit, that original shit doesn't look quite so bad in comparison. So take your shit monolith, surround it by shit services that distributed it across a shitty network, and now your original self inflicted shit design is just 1/3rd of the shit you gotta deal with. Totally not as bad as it used to be!

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#65

Post by ldargin »

LVB wrote: post #37

> Moving the solution to Amazon EC2 and Amazon ECS also allowed us to use the Amazon EC2 compute saving plans that will help drive costs down even further. So various parts of Amazon have to work through the AWS same pricing programs that the rest of us do?

Yes, to keep track of costs.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#66

Post by cogitoergofutuo »

bberrry wrote: post #26

I wouldn't call it a monolith as the number of instances could be scaled up. Mono implies single instance. They just combined multiple microservices into a larger one.

It’s also not really serverless to begin with, because at the end of the day code is being executed on a physical device that many of us might call a “server”

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#67

Post by oaiey »

silisili wrote: post #46
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I absolutely agree, buuuut also realize we as programmers don't even have the same definition of what a microservice is. A lot of people here say...one service per team. But to me that is, or can be, a monolith. Often a team is a product line, so you have one service for that product. Is that a monolith? I don't know either, I guess. I -do- know most people who go around promoting that sweet microservice life end up…

Yes! It is about splitting your business in reasonable chunks/domain/boundaries. Pizza rules/rewrite rules ... that is all nonsense.

But doing it right, is nevertheless hard. Because cutting your business into chunks ... is not as easy as it always looks.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#68

Post by lastangryman »

My word. I'm sort of gob smacked this article exists.

I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another poster commented, I wouldn't be surprised if it's taken down at some point.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#69

Post by charcircuit »

mjr00 wrote: post #62
wg0 wrote: post #20

Earlier quoted context omitted.

OR each team can write their features as a python package, rust crate, go module to be included in the main app. Libraries, versioned.

And then team 1 needs to upgrade pandas to 2.0, but team 2 is still on pandas 1, so when the main app pulls them in nothing works, so you need to start a cross-team committee to schedule the work to upgrade a single library... Separate services aren't a silver bullet, but as an fyi to the younger software developers, we tried "just have all teams work on the same code base and deployable artifact" for a long while an…

All the teams will need to migrate sooner or later so figuring out all of the potential problems in migrating and having everyone do it an once is more efficient than each team needing to figure it out separately.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#70

Post by oaiey »

jjevanoorschot wrote: post #57

The title is editorialised to be clickbait. The original title is "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%". They changed a single service, the Prime Video audio/video monitoring service, from a few Lambda and Step Function components into a 'monolith'. This monolith is still one of presumably many services within Prime Video.

The worth here is that Amazon is writing about not going into AWS PaaS native programming (what Lambda is) because it is too expensive for them.

That has some newsworthiness and the title kind of reflects that.

Post reply on HN