Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

361–370 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#361

Post by amoss »

mehdix wrote: post #84
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Once I was called to a meeting in a sibling department as a cloud advisor. They wanted to migrate to AWS cloud. The conversation went as below. - Does your app work fine? - Yes. - Do you have any problems? - No. - Why do you want to migrate then? - Silence.

The CFO read an article in Forbes that said we would save money by migrating to the cloud, so we now have an unlimited budget for consultants to build us a cloud platform....

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#363

Post by pjmlp »

The_Colonel wrote: post #52
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

The problem is that we are now in the golden age of Web services, where everything is headless and controlled via API, so the business logic plugging all those APIs need to live somewhere.

Naturally it could be a single container taking care of all those integrations.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#364

Post by Guid_NewGuid »

alpos wrote: post #318

"We built a video stream processor by splitting every 1080p+, multi hour long, 30-60fps video into individual images and copying them across networks multiple times." Not surprising that didn't go will. This strikes me as a punching bag example. Anyone who has worked with images, video, 3d models, or even just really large blocks of text or numbers before (any kind of actually "big data") knows how much work goes int…

To be fair it is somewhat a punching bag example but I think what people are reacting to, but maybe not articulating well, is the presumption for microservices by the powers-that-be.

Of course the only rational take on monoliths versus microservices is "use the right tool for the job".

But systems design interviews, FAANG, 'thought leaders', etc basically ignore this nuance in favour of something like the following.

Question: design pastebin (edit, I of course mean a URL shortener not pastebin)

Rational first pass but wrong Answer: Have a monolith that chucks the URL in the database.

Whereas the only winning answer is going to have a bunch of services, separate persistence and caching, a CDN, load balancing, replicas, probably a DNS and a service mesh chucked in for good measure.

I think this article shows that this is training and producing people who can't even think of the obvious first answer they have been so thoroughly indoctrinated.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#365

Post by boxed »

mike_hearn wrote: post #239
boxed wrote: post #112

Earlier quoted context omitted.

I built a little microservice on the side of my monolith for PDF creation. It used headless chrome and ghostscript to render html to a nice PDF. The problem I had with having that code inside the monolith was that it increased my docker image creation for deploys by a lot . And that code pretty much never changed anyway. I did feel a bit embarrassed having to make a microservice after having argued against them so mu…

Doesn't Docker use cached image layers to solve that? Your PDF rendering could be in one layer that never changes, and the rest goes on top.

In theory yes, but that doesn't work well on github actions for example. The cache layering seems quite bad there for some reason.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#366

Post by adql »

moonchrome wrote: post #271
ojkelly wrote: post #254

Earlier quoted context omitted.

But, by omission is reads that Prime Video rebuilt their stack without serverless and got a 90% cost reduction. This post is going to pick up a lot of traction and I suspect these comments are going to bikeshed monolith vs microservices for the next day. On reading it, this is for a video quality monitoring system, that needs to consume and process video. Generally a compute and time intensive task. Something not alw…

Serverless has such bullshit insidious pricing that makes it seem like you're saving money only to figure out you're in shit once you're knee deep in it. For example you'll have to read fine print to find out that 256MB lambda will have the compute power of a 90s desktop PC because compute scales with memory. And to get access to "one core" of compute you have to use like 2GB of memory. Now you may say "serverless is…

well, they have no incentive to not make you pay for CPU time on your application startup

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#367

Post by deterministic »

Micro-services is BS invented by cloud providers to solve problems you don’t have at 10x the cost.

The worst software systems I have ever seen were micro-services. One of them is more than 20 years old. The WTF count per minute is exponential.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#368

Post by terom »

> The second cost problem we discovered was about the way we were passing video frames (images) around different components. To reduce computationally expensive video conversion jobs, we built a microservice that splits videos into frames and temporarily uploads images to an Amazon Simple Storage Service (Amazon S3) bucket. Defect detectors (where each of them also runs as a separate microservice) then download images and processed it concurrently using AWS Lambda. However, the high number of Tier-1 calls to the S3 bucket was expensive.

Taking "malloc for the Internet" [1] a bit /too/ literally there.

[1] https://aws.amazon.com/blogs/aws/eight-years-and-counting-of...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#369

Post by adql »

LASR wrote: post #274

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed". Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. My candid reaction: Seriously? WTF? I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and…

> This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed".

I feel that's like 95% of the "we migrated from X to Y and now it is better"; most of improvements coming from rewriting app/infrastructure after learning the lessons with only small part sometimes being the change in tech

Post reply on HN