Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

251–260 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#251

Post by motbus3 »

goodpoint wrote: post #245
motbus3 wrote: post #156

Earlier quoted context omitted.

I think it is fine. There are scenarios were you need distributed and there are scenarios that you don't. IMO, distributed software is more practical for working development than for technical reasons. We all know from basic stuff that performing software comes from single structures that does not require packing and unpacking data But scaling large applications is hard, and it was much more expensive back then. Now…

[flagged]

I do not care about the downvoting it is unfortunate that there is no comment back saying why the person disagrees

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#252

Post by bottlepalm »

arp242 wrote: post #209
kubanczyk wrote: post #198

Earlier quoted context omitted.

Two green changes merged to a green main can produce a red main. This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default: > Require branches to be up to date before merging > Whether pull requests targeting a matching branch must be tested with the latest code.

> Two green changes merged to a green main can produce a red main. I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.

It's definitely an issue if you're treating main as 'good to deploy'. Also with more devs the chance of this happening is pretty good. Definitely want to enforce a branch is up to date before allowing merge (though with a lot of devs this can become difficult) not sure if there is some sort of merge/test pipeline solution out there.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#253

Post by IshKebab »

YetAnotherNick wrote: post #114
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

I think the trust issue doesn't really justify microservices. Assuming everyone is using interoperable languages you can still have a monolith with clear API contacts and separate ownership by using traditional libraries.

That is a good point about reliability and cost though. I hadn't heard that before.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#254

Post by ojkelly »

debdut wrote: post #105
iamflimflam1 wrote: post #50

This really is a click bait title. They are talking about their video quality monitoring service, not their video streaming service. It’s something they use to check for defects in the video stream - hence the storing of individual frames in S3. Original title: Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%

The subtitle is "The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs." And the article itself mentions the 90% cost reduction. So the title seems pretty much in-line with the original intent.

But, by omission is reads that Prime Video rebuilt their stack without serverless and got a 90% cost reduction.

This post is going to pick up a lot of traction and I suspect these comments are going to bikeshed monolith vs microservices for the next day.

On reading it, this is for a video quality monitoring system, that needs to consume and process video. Generally a compute and time intensive task. Something not always suited to severless, particularly when it’s not easy to parallelise.

The task at hand doesn’t sound ideally suited to serverless, but the existence of the post shows that’s not readily obvious. So it’s a valuable post to explain a scenario where a few big machines is the best call.

But the sensationalism of the headline, would suggest all serverless is expensive and wasteful. When in reality the same is true for a non-ideal workload on a monolith.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#255

Post by lastangryman »

iLoveOncall wrote: post #238
lastangryman wrote: post #143

Earlier quoted context omitted.

To most people, "mono" refers to a single codebase, not a single deployed instance. I've worked on many monoliths that run multiple instances in production. Microservices are no more or less scalable than a monolith. The main benefit of Microservices is allowing multiple teams to work independently from each other without everyone "stepping on each others toes". You can have scalable monoliths and unscalable microser…

> Microservices are no more or less scalable than a monolith. This is not fully true. A microservice architecture is more finely scalable than a monolith. To take a very basic example, if you have a peak of users watching a video you can scale up the microservice dedicated to serving videos, but not scale up the service dedicated to users signups, which isn't having an increased load.

> A microservice architecture is more finely scalable than a monolith.

Apologies, but I strongly disagree and I'm going to go on a bit of a rant here....

This is a myth, and one of the reasons people are making these ridiculous architecture descisions. If you have a monolith that serves videos and enables signups, you can deploy as many instances of that as you like based on the highest need. It doesn't matter if user signups are a fraction of video watches, it just means that your user signup endpoint is not getting called as much. Maybe you're deploying a larger codebase than you need to but that's hardly a downside.

In your example, let's say we have 2 endpoints that are behind a gateway or L7 LB so that we can point them at different codebases if we like:

- videoservice.com/signup

- videoservice.com/watch

If I'm geting 100k rps to /watch, and 100 rps to /signup, I can just deploy loads of instance of my monolith behind the /watch endpoint. Maybe that monolith contains code for /signup, but it's not going to get called. So what.

I've seen this approach used in many places. You don't need to split the code to do this at all. Sure it might feel "cleaner" to you to do this, but it's not needed.

Now, you may get to a point where your deployment is really heavy and time consuming and you don't want to deploy everything just to scale up /watch - but again I'd argue that is not really anything to do with scalability, it's about being able to deploy things independently. Using a microservice doesn't make your service more scalable here, but it might make it easier to deploy.

Microservices are nothing to do with scalability. They are about how you organise code and teams to achieve better development velocity.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#256

Post by dtech »

The_Colonel wrote: post #130
rostos wrote: post #82

Earlier quoted context omitted.

That organizational/architecture benefit is strong though.

I would frame it as a necessity rather than a benefit. Having siloed teams (services) is usually a problem which is better to avoid as much as you can.

Having autonomous teams is great for scaling and allowing everyone to go fast, without teams constantly blocking each other.

Having hundreds of engineers work in a single monolith in a single repo without any kind of (enforced) boundaries is a one way ticket to a big ball of mud. You need to invest heavily in tooling to make it work, and e.g. Google does so.

Having a network in between teams is a relatively easy way to enforce boundaries.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#257

Post by goodpoint »

sgtnoodle wrote: post #63
ad-astra wrote: post #7

Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.

Indeed, it does seem rather ridiculous at face value. On the other hand, I have coworkers that run CPU-IPC bound workloads inside x86-64 docker containers on M1 macs (incurring the overhead of both machine code emulation and OS virtualization). I have other coworkers sweating for hours whether to use 32-bit or 64-bit integers for APIs designed for microcontrollers running at 300Mhz. I have even more coworkers writing…

> rust because it's "memory safe" and "so fast", but they have no idea that they're doing

That summarizes hype-based design very well.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#259

Post by throwaway2990 »

mparnisari wrote: post #92

As a former AWS employee I can almost guarantee that the person that made the original design got a promotion over it.

As a never been AWS employee I can almost guarantee you the original design was most likely simple and the use of lambdas and step functions a good choice and not expensive but the functionality grew and the cost sky rocketed. This is only normal evolution of a service.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#260

Post by 123pie123 »

mehdix wrote: post #84
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Once I was called to a meeting in a sibling department as a cloud advisor. They wanted to migrate to AWS cloud. The conversation went as below. - Does your app work fine? - Yes. - Do you have any problems? - No. - Why do you want to migrate then? - Silence.

because they want "cloud" on ther Resume/ CV
Post reply on HN