Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

171–180 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#171

Post by bagels »

sgtnoodle wrote: post #89
selcuka wrote: post #25

Earlier quoted context omitted.

They should've serialised bitmaps to JSON and used SQS instead. /s

Over 15 years ago now, I was an intern at Toyota. We were working with an in-house python based framework for doing cool/terrible drive-by-wire things with test cars. I had a project to work around a bottleneck of the framework. It could only process about 70 CAN frames per second before running out of CPU. The vehicle's CAN bus had several thousand per second, though. At the time I was able to fix the problem by add…

Also ctypes

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#172

Post by foepys »

boxed wrote: post #112
The_Colonel wrote: post #52

Earlier quoted context omitted.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

I built a little microservice on the side of my monolith for PDF creation. It used headless chrome and ghostscript to render html to a nice PDF. The problem I had with having that code inside the monolith was that it increased my docker image creation for deploys by a lot . And that code pretty much never changed anyway. I did feel a bit embarrassed having to make a microservice after having argued against them so mu…

That's exactly what my team did at a former company. We generated reports through a legacy document engine because some customers cannot switch/update their report templates and so we moved the logic out of the monolith into a service to get rid of a large portion of our dependencies.

Moved the monolith to .NET Core, kept the report service on .NET Framework. A win for everybody.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#173

Post by bottlepalm »

YetAnotherNick wrote: post #114
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

The problem is when the company re-orgs you can easily end up with many more microservices than there are teams. Then you’re really up a creek.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#174

Post by smitty1e »

Two naive ideas that may be OK as a going-in position:

- granularity

- bandwidth negligibility

Breaking everything down to a gnat's ass might improve testability, but is testability the product? Do I really need a Java stack trace that reads like an Andrew Wiles proof?[1] Maybe I do, at scale.

Then there is the non-zero cost of the packet shuffling. Every edge in the aechitctural graph, not just the nodes, costs. But we just throw a waiter into the code and move on to the next line. No biggie.

What was most interesting was "It also increased our scaling capabilities." Granularity was supposed to let "serverless" absorb the entire universe, I thought.

At a higher level of abstraction, maybe The Famous Article is a map/reduce job: the requirements dissolved into solution, and a proper number of components precipitated out.

[1] https://en.m.wikipedia.org/wiki/Wiles%27s_proof_of_Fermat%27...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#175

Post by withinboredom »

Hamuko wrote: post #160
Aeolun wrote: post #131

Earlier quoted context omitted.

I feel like it’s an object lesson in using the right solution for a problem. Step functions do not appear to me to be something that you’d use for things that need to be executed multiple times per second.

Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in…

If you already have those instances for something else, there's rarely a good reason not to cohost. Save $2 a month and the maintenance burden.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#176

Post by Hamuko »

raxxorraxor wrote: post #169
Hamuko wrote: post #160

Earlier quoted context omitted.

Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in…

For some Alexa integrations it is neat and convenient, but I went back to hosting such small interfaces as a service on another server. Not an EC2 instance, just another hosted unmanaged server. They are as cheap as they can get right now.

I'm not really interested in the operational overhead that brings for these small services. Cost-wise they might be just about neck-and-neck, but at least I don't need to worry about the server going down, or having outdated software. Lambda gives me scaling, load balancing and redundancy for that $2.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#177

Post by bottlepalm »

__turbobrew__ wrote: post #140
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

Tests passing should be a gate to merge into main. Tests can also be run in parallel.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#178

Post by Alacart »

Hamuko wrote: post #160
Aeolun wrote: post #131

Earlier quoted context omitted.

I feel like it’s an object lesson in using the right solution for a problem. Step functions do not appear to me to be something that you’d use for things that need to be executed multiple times per second.

Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in…

In your example, you compare Lambda against a separate monolith for handling the webhooks, but with a monolith wouldn't the comparison be between lambda and just adding a route and controller (or equivalent) to the monolith?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#179

Post by bottlepalm »

silisili wrote: post #46
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I absolutely agree, buuuut also realize we as programmers don't even have the same definition of what a microservice is. A lot of people here say...one service per team. But to me that is, or can be, a monolith. Often a team is a product line, so you have one service for that product. Is that a monolith? I don't know either, I guess. I -do- know most people who go around promoting that sweet microservice life end up…

Every service needs it’s own database. Then you need to handle all the DLQ errors between services as well.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#180

Post by withinboredom »

__turbobrew__ wrote: post #140
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.
Post reply on HN