Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

81–90 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#81

Post by NBJack »

freeone3000 wrote: post #61
kristopolous wrote: post #55

Earlier quoted context omitted.

There's a bunch of things I'd like to have them do. If they could span across machines like clusters that would be amazing. If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing. If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool. If I could…

Your asks seem easily answered with docker + kubernetes. Actually, this is in fact the use case for kubernetes — a fault tolerant distributed system running arbitrarily, simply packaged code. This has to be what you’ve tried — what issue are you running into?

Kubernetes isn't something I'd put in the same sentence as 'easy'. Docker is a close contender for the same.

I still recall the day when my local Docker builds necessitated a new router to properly manage streaming traffic at home while I downloaded a few GBs of layer images. Or the time I wanted to setup a 'simple' hosted Kubernetes cluster of my own in my lab for testing, only to discover the nightmare that is networking on it. Then there was the grim discovery that Docker containers were much more sensitive to the hosting environment than I had assumed, resulting in some fun "but it worked and tested fine" moments.

Did they all work eventually for me? Yes. Was it simple? Not by my standards.

If you find all that simple, kudos to you.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#82

Post by rostos »

The_Colonel wrote: post #52
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

That organizational/architecture benefit is strong though.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#83

Post by ocdtrekkie »

bagels wrote: post #45
ocdtrekkie wrote: post #13

Earlier quoted context omitted.

Considering they don't actually pay the bill for this and it is internal accounting, probably not. Belt tightening has probably pushed cloud providers to figure out if they're wasting stuff they could put to better use, and I assume when it launched and nobody was watching Prime Video, inefficiencies were both smaller and less noticeable.

Their team/org has resource budgets too.

Oh, absolutely. This makes the Prime Video team look more profitable on paper. But also all streaming services were pretty much launched with an expectation of taking losses for years, so Prime Video being expensive doesn't look unusual for a while. And since it's an internal cost, it's not actually Amazon paying someone, there's really not a significant reason for someone outside the Prime Video team to say "hey, you're too big of an AWS customer".

More than likely, Prime Video making their numbers look better makes AWS' numbers look (slightly) worse, because they're doing a little less business. In the overarching grand scheme of things, this will save Amazon some amount of physical computing resources they weren't getting paid by an outside customer for, but good luck figuring out how much that actual real world savings is.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#84

Post by mehdix »

bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Once I was called to a meeting in a sibling department as a cloud advisor. They wanted to migrate to AWS cloud.

The conversation went as below.

- Does your app work fine?

- Yes.

- Do you have any problems?

- No.

- Why do you want to migrate then?

- Silence.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#85

Post by mlhpdx »

yelnatz wrote: post #53
ad-astra wrote: post #7

Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.

How would you do it?

Lots of opportunities short of rearchitecting: use batching; use multi threading in lambdas; use S3 range requests; use the EXPRESS execution model; etc, etc

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#86

Post by henry_bone »

The_Colonel wrote: post #52
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

Seems like Conway's law applies:-

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." -- Melvin E. Conway

https://en.wikipedia.org/wiki/Conway%27s_law

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#87

Post by nicoco »

vivegi wrote: post #34

Of all the video streaming services I have used, PrimeVideo is the one where the video/audio sync becomes terrible progressively. It is pretty bad. It happens in 8 out of 10 movies. There is some misconfiguration in their AV transcoding pipeline. And here, we have an article talking about Monolith vs. Microservices improving user experience.

Nevers had any issue with bittorrented AWS.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#88

Post by jddj »

kristopolous wrote: post #55
wg0 wrote: post #14

Earlier quoted context omitted.

As for me,I have been trying to discover that 5% that cannot be done without microservices.

There's a bunch of things I'd like to have them do. If they could span across machines like clusters that would be amazing. If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing. If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool. If I could…

Did you ever play with the Erlang VM?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#89

Post by sgtnoodle »

selcuka wrote: post #25
ad-astra wrote: post #7

Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.

They should've serialised bitmaps to JSON and used SQS instead. /s

Over 15 years ago now, I was an intern at Toyota. We were working with an in-house python based framework for doing cool/terrible drive-by-wire things with test cars.

I had a project to work around a bottleneck of the framework. It could only process about 70 CAN frames per second before running out of CPU. The vehicle's CAN bus had several thousand per second, though. At the time I was able to fix the problem by adding filtering to the CAN adapter's kernel module.

A couple years later, I worked on replacing the python based framework with C++. I discovered the underlying root cause of the bottleneck. Someone (cough my manager) had figured out a very "pythonic" way to extract bit-packed fields from the 64-bit CAN frame payloads. They converted every 8-byte payload buffer into a canonical binary representation, i.e. ascii strings of 1's and 0's. They then used string slicing syntax to extract fields. Finally, they casted the resulting substrings back to integers. Awesome!

I've since used python many times to process CAN frames in realtime, scaling up to thousands of frames per second without the CPU breaking a sweat. One trick is to use integer bit shifts and masks rather than string printing, slicing and parsing...

Post reply on HN