Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

401–410 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#401

Post by boudin »

ojkelly wrote: post #254
debdut wrote: post #105

Earlier quoted context omitted.

The subtitle is "The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs." And the article itself mentions the 90% cost reduction. So the title seems pretty much in-line with the original intent.

But, by omission is reads that Prime Video rebuilt their stack without serverless and got a 90% cost reduction. This post is going to pick up a lot of traction and I suspect these comments are going to bikeshed monolith vs microservices for the next day. On reading it, this is for a video quality monitoring system, that needs to consume and process video. Generally a compute and time intensive task. Something not alw…

I don't get the microservice to monolith part of this blog post neither.

It does look that they replaced the serverless implementation of a service with an hosted app because this service wasn't scaling.

They don't really communicate around the architecture of the whole Prime Video product but it doesn't look like a monolith.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#402

Post by adql »

jabradoodle wrote: post #359
awaythrow98765 wrote: post #289

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Cloud vendors also mostly sell minimum use packages for discounts in the range of 20 to 80% (called e.g. "committed use discount" or "compute savings plan"). Lots of businesses use those, because two-digit discou…

It's not really the same spot in that your paying monthly rather than upfront. Devs tend to think about total $, the business/accountants do care about Opex vs Capex. Also it's going to be simpler to provision your base (commited use) on the cloud and then handle bursts on the cloud, than it is to have your base on prem and burst to the cloud.

> It's not really the same spot in that your paying monthly rather than upfront. Devs tend to think about total $, the business/accountants do care about Opex vs Capex.

You can buy physical servers in leasing ,turning it into opex

You can also rent them for little bit extra via managed dedicated servers from vendors like OVH.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#403

Post by andersa »

boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

It is even more amazing when the entire $10k AWS setup can be replaced by a single minimally optimized monolith running on one $20/month Hetzner server that responds several times faster to most requests due to no internal latency.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#404

Post by NohatCoder »

andai wrote: post #353
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

I haven't dealt with high traffic systems but isn't a few requests per second well within the capabilities of a $5 VPS?

A $5 instance gets you something like 1 core on a slightly dated CPU. Aka approximately as much processing power as a top of the line desktop processor 15 years ago. Aka enough processing power to fill a 25 Gb/s port with TLS data (not that you have the port to go with that).

A few requests per millisecond should be well within the capabilities of this instance, depending on the complexity of each request of course.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#405

Post by burnished »

iLoveOncall wrote: post #199
iamflimflam1 wrote: post #50

This really is a click bait title. They are talking about their video quality monitoring service, not their video streaming service. It’s something they use to check for defects in the video stream - hence the storing of individual frames in S3. Original title: Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%

Yes this is a ridiculous clickbait. For once the original title is not and the poster had to make it so... Why is dang not changing it back? PrimeVideo is very much based on a microservice architecture. Hell, my team which isn't client facing and has a very dedicated purpose has easily more microservices than engineers.

Well, it was probably the middle of the night.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#406

Post by datadeft »

moonchrome wrote: post #347
datadeft wrote: post #293

Earlier quoted context omitted.

All of things you are complaining about are well known facts that are clearly stated in the documentation. I don't care about what is the equivalent computing power in 90s desktop measurement because you cannot replace a lambda function with a 90s desktop, so it is pointless. The right approach is: I have a problem A that I can implement using AWS Lambda, AWS EC2 or your favourite DHH approved stack, how much of thes…

Can you point me to where this is clearly stated in the documentation ? I only found one reference as a passing note when I went searching for it. This would be a value displayed in the pricing calculator with a link to explanation if they were being honest. 90s CPU comparison is just to demonstrate how out of place it is with what people are used to even on lowest tier hosts with shared CPU cores. Low ram compute se…

Memory and computing power

Memory is the principal lever available to Lambda developers for controlling the performance of a function. You can configure the amount of memory allocated to a Lambda function, between 128 MB and 10,240 MB. The Lambda console defaults new functions to the smallest setting and many developers also choose 128 MB for their functions.

https://docs.aws.amazon.com/lambda/latest/operatorguide/comp...

CPU Allocation

It is known that at 1,792 MB we get 1 full vCPU1 (notice the v in front of CPU). A vCPU is “a thread of either an Intel Xeon core or an AMD EPYC core”2. This is valid for the compute-optimized instance types, which are the underlying Lambda infrastructure (not a hard commitment by AWS, but a general rule).

If 1,024 MB are allocated to a function, it gets roughly 57% of a vCPU (1,024 / 1,792 ~= 0,57). It is obviously impossible to divide a CPU thread. In background, AWS is dividing the CPU time. With 1,024 MB, the function will receive 57% of the processing time. The CPU may switch to perform other tasks on the remaining 43% of the time.

The result of this CPU allocation model is: the more memory is allocated to a function, the faster it will accomplish a given task.

https://dashbird.io/knowledge-base/aws-lambda/resource-alloc...

> For reference I have a devops team in-company that deployed and maintained several AWS projects, including some serverless

Same here.

> even they were surprised at the low compute available at low RAM lambdas.

I wasn't because we measured it and based on the measurement we calculated what we want. I think it is a good approach not to assume anything.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#407

Post by sorcix »

latexr wrote: post #216
nomilk wrote: post #192

Earlier quoted context omitted.

I had to google what 'cargo culting' meant. But I laughed when I found out. https://en.wikipedia.org/wiki/Cargo_cult_programming#:~:text... .

Do you have a link to documentation on that method of highlighting text? Hard to search for due to the all the non-letter characters.

https://chromestatus.com/feature/4733392803332096

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#408

Post by nikanj »

boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

The thing could run on a $99 Hetzner box just fine, but that looks terrible on your CV

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#409

Post by dijit »

hobs wrote: post #397
NohatCoder wrote: post #350

Earlier quoted context omitted.

This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al. A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.

I just spent the better part of two years advocating, pushing, and fighting for months to add new bandwidth to our datacenter. Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other. While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a mi…

I counter your anecdote with mine!

A cloud vendor (who will be nameless as I signed an NDA specifically that prevents me from disparaging them; but one of the big three) ran out of capacity for me and it was 3 months before they managed to fix it. -- that was with a couple million a month in spend.

Cloud is still servers; you just depend on someone elses capacity management skills and you hope that there isn't a rush to populate a location (like when a region goes down and everyone's auto-provisioners move regions to yours)

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#410

Post by whstl »

citrin_ru wrote: post #123
rostos wrote: post #82

Earlier quoted context omitted.

That organizational/architecture benefit is strong though.

It’s depends on how you slice services. For one micro-service per a team I see benefits. On another end of the spectrum a single team managing 10-20 micro-services with more than one service per developer. IMHO it creates more problems than solves. Also it is usually a waste of HW resources because a library call it is cheaper then a network request.

I think it depends on more than that.

A single team managing 10 microservices that actually make sense to be microservices (like the PDF renderer example above [1]) is kinda good and perfectly manageable.

A team with one single microservice that would actually work better if it was part of a monolith is already in the "creates more problems than it solves" territory.

[1] https://news.ycombinator.com/item?id=35812294

Post reply on HN