Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

261–270 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#261
post #216
post #192

Earlier quoted context omitted.

I had to google what 'cargo culting' meant. But I laughed when I found out. https://en.wikipedia.org/wiki/Cargo_cult_programming#:~:text... .

Do you have a link to documentation on that method of highlighting text? Hard to search for due to the all the non-letter characters.

Text fragments: https://wicg.github.io/scroll-to-text-fragment/

Currently supported on all non-Firefox major browsers. https://caniuse.com/url-scroll-to-text-fragment

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#262

Earlier quoted context omitted.

Tests passing should be a gate to merge into main. Tests can also be run in parallel.

Two green changes merged to a green main can produce a red main. This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default: > Require branches to be up to date before merging > Whether pull requests targeting a matching branch must be tested with the latest code.

> Require branches to be up to date before merging

Because this really doesn't scale well with the number of developers.

For larger teams the solution is to use a merge queue, e.g. https://shopify.engineering/successfully-merging-work-1000-d...

I haven't tried it, but GitHub is now offering such feature in public-beta: https://github.blog/changelog/2023-02-08-pull-request-merge-...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#263

Earlier quoted context omitted.

Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)

If you're sensitive to a 25% performance improvement, why not switch languages and get a 1000% performance improvement?

An easily tested compatible upgrade that gets a free performance boost… vs lots of engineering effort to rewrite… yeah that’s just not going to fly with management. Who are probably looking at the 25% performance boost as a 20% cost reduction not a 20% speed increase

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#264

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

I think the trust issue doesn't really justify microservices. Assuming everyone is using interoperable languages you can still have a monolith with clear API contacts and separate ownership by using traditional libraries. That is a good point about reliability and cost though. I hadn't heard that before.

What if some team makes the their part 10 times slower. This is not a theoretical scenario, but one I saw happening many times. While technically you could partition and monitor each part of the monolith separately, but then you are just reinventing microservice architecture.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#265

Earlier quoted context omitted.

To most people, "mono" refers to a single codebase, not a single deployed instance. I've worked on many monoliths that run multiple instances in production. Microservices are no more or less scalable than a monolith. The main benefit of Microservices is allowing multiple teams to work independently from each other without everyone "stepping on each others toes". You can have scalable monoliths and unscalable microser…

> Microservices are no more or less scalable than a monolith. This is not fully true. A microservice architecture is more finely scalable than a monolith. To take a very basic example, if you have a peak of users watching a video you can scale up the microservice dedicated to serving videos, but not scale up the service dedicated to users signups, which isn't having an increased load.

> you can scale up the microservice dedicated to serving videos, but not scale up the service dedicated to users signups, which isn't having an increased load.

No, splitting a codebase does not magically make it more scalable in production. You still have to prove that the authentication component would create significant unnecessary load if it was scaled up together with the video service.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#266
post #126

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

Maybe they'll publish the opposite results in 6 months

And someone else will get promoted.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#267

Earlier quoted context omitted.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution. > Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employe…

What if you have a really big database? Still 2minutes?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#268

Breaking things into tiny functions and putting them on many different servers incurs tradeoff costs in both complexity and compute. There is a complexity cost in having to deal with the setup, security, and orchestration of those functions, and a compute cost because if the overall system is running constantly it will be less efficient and therefore more expensive than running on one box.

I agree on the tradeoffs you have to make. The main cost driver here was storage and traffic, though.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#269

Earlier quoted context omitted.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#270
post #112

Earlier quoted context omitted.

I built a little microservice on the side of my monolith for PDF creation. It used headless chrome and ghostscript to render html to a nice PDF. The problem I had with having that code inside the monolith was that it increased my docker image creation for deploys by a lot . And that code pretty much never changed anyway. I did feel a bit embarrassed having to make a microservice after having argued against them so mu…

Doesn't Docker use cached image layers to solve that? Your PDF rendering could be in one layer that never changes, and the rest goes on top.

The problem is when you have more than one such service. Now when one of them changes, all of them need to be rebuilt. You can solve this with multi-stage builds, but those only work if your build result can be easily copied.
Post reply on HN