Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

121–130 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#121

Post by 12907835202 »

The_Colonel wrote: post #52
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

Depends if you're counting all the WordPress level projects. Then it'd be 0.01% Vs 99.99% I'd imagine.

But that just puts into perspective how silly this argument is because I have no idea what a project means to other people.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#122

Post by jjevanoorschot »

kstenerud wrote: post #94
jjevanoorschot wrote: post #57

The title is editorialised to be clickbait. The original title is "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%". They changed a single service, the Prime Video audio/video monitoring service, from a few Lambda and Step Function components into a 'monolith'. This monolith is still one of presumably many services within Prime Video.

The subtitle is "The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs." And the article itself mentions the 90% cost reduction. So the title seems pretty much in-line with the original intent.

The title makes it sound like Prime Video abandoned microservices all-together, but in reality they only did so for a single service.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#123

Post by citrin_ru »

rostos wrote: post #82
The_Colonel wrote: post #52

Earlier quoted context omitted.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

That organizational/architecture benefit is strong though.

It’s depends on how you slice services. For one micro-service per a team I see benefits. On another end of the spectrum a single team managing 10-20 micro-services with more than one service per developer. IMHO it creates more problems than solves. Also it is usually a waste of HW resources because a library call it is cheaper then a network request.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#124

Post by alexchamberlain »

Somewhat interesting article, but this isn't a monolith, at least not by a microservice fanboy definition.

The product (Prime Video) is still built using many business oriented services. Furthermore, this service appears to be developed and operated by a single team.

That being said, there are some lessons here - there are good ideas in most design paradigms, but if you take them to the extreme, you're going to see some weird side effects. Understand the benefits and engineer a balanced solution.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#125

Post by mpweiher »

bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Yes.

However, I think there is something hiding inside the µservices movement that is actually much more generally applicable and useful: API-first development.

And of course good old OO.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#126

Post by seydor »

lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

Maybe they'll publish the opposite results in 6 months

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#127

Post by simplotek »

supriyo-biswas wrote: post #42
dpwm wrote: post #16

Earlier quoted context omitted.

> We designed our initial solution as a distributed system using serverless components (for example, AWS Step Functions or AWS Lambda), which was a good choice for building the service quickly. The message seems more that they outgrew AWS lambda but that lambda was a good choice at first.

The post literally says that they could hit only 5% of the expected workload with their server less architecture, so IMO it is still quite negative.

> The post literally says that they could hit only 5% of the expected workload with their server less architecture, so IMO it is still quite negative.

Emphasis on "their server less architecture". Sometimes good tools are used poorly.

For example they describe a high throughout workload, and each workload spread through a bunch of lambdas that handled bite size bits of the workflow. Also, they managed the workflow with step functions. Just imagine the number of network calls involved to run a single job, let alone all the work pulling data to/from a data store like S3 into/out of a lambda. I'd guess the bulk of their wall time was IO to setup the computation.

Of course you get far better performance if you get rid of all these interfaces.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#128

Post by oblio »

est wrote: post #100
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

> make sense 5% of the time Micro-services were invented by an outstanding software outsourcing company to milk billable hours and offload responsibilities in large org. If you want to save cost and not-that-large business, go for monolith-first. Keep it modular.

Which company? Thoughtworks?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#129

Post by deanCommie »

supriyo-biswas wrote: post #42
dpwm wrote: post #16

Earlier quoted context omitted.

> We designed our initial solution as a distributed system using serverless components (for example, AWS Step Functions or AWS Lambda), which was a good choice for building the service quickly. The message seems more that they outgrew AWS lambda but that lambda was a good choice at first.

The post literally says that they could hit only 5% of the expected workload with their server less architecture, so IMO it is still quite negative.

5% of the expected workload for a Prime Video service is probably more than 99% of the workloads of the readers.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#130

Post by The_Colonel »

rostos wrote: post #82
The_Colonel wrote: post #52

Earlier quoted context omitted.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

That organizational/architecture benefit is strong though.

I would frame it as a necessity rather than a benefit. Having siloed teams (services) is usually a problem which is better to avoid as much as you can.
Post reply on HN