Scaling up the Prime Video audio/video monitoring service and reducing costs • Hacker News Board

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#291

Post by Akronymus » Thu, May 04, 2023, 10:06 AM UTC

anonzzzies wrote: post #269

withinboredom wrote: post #180

Earlier quoted context omitted.

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.

Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#292

Post by manfre » Thu, May 04, 2023, 10:06 AM UTC

tough wrote: post #267

kerkeslager wrote: post #242

Earlier quoted context omitted.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution. > Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employe…

What if you have a really big database? Still 2minutes?

I'm intentionally ignoring any of the sarcasm in your comment. The time needed for a db dump is always dependent upon the amount of data. This is true regardless of the db software or where it's running.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#293

Post by datadeft » Thu, May 04, 2023, 10:06 AM UTC

moonchrome wrote: post #271

ojkelly wrote: post #254

Earlier quoted context omitted.

But, by omission is reads that Prime Video rebuilt their stack without serverless and got a 90% cost reduction. This post is going to pick up a lot of traction and I suspect these comments are going to bikeshed monolith vs microservices for the next day. On reading it, this is for a video quality monitoring system, that needs to consume and process video. Generally a compute and time intensive task. Something not alw…

Serverless has such bullshit insidious pricing that makes it seem like you're saving money only to figure out you're in shit once you're knee deep in it. For example you'll have to read fine print to find out that 256MB lambda will have the compute power of a 90s desktop PC because compute scales with memory. And to get access to "one core" of compute you have to use like 2GB of memory. Now you may say "serverless is…

All of things you are complaining about are well known facts that are clearly stated in the documentation.

I don't care about what is the equivalent computing power in 90s desktop measurement because you cannot replace a lambda function with a 90s desktop, so it is pointless.

The right approach is: I have a problem A that I can implement using AWS Lambda, AWS EC2 or your favourite DHH approved stack, how much of these cost compare to each other.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#294

Post by adql » Thu, May 04, 2023, 10:08 AM UTC

dragonwriter wrote: post #136

lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

> Which is good lesson, and a good story, but there’s a kind of irony it’s come from an internal Amazon team. As another poster commented, I wouldn’t be surprised if it’s taken down at some point. Why? Using the model they switched to (which uses a different set of AWS services) instead of the model they switched from is a recommendation that the AWS tech advisers that are made available to enterprise customers will…

The "different set of services" is so basic it can be ran anywhere, and near-everywhere is cheaper than AWS

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#295

Post by datadeft » Thu, May 04, 2023, 10:09 AM UTC

yazaddaruvala wrote: post #230

dbish wrote: post #147

Earlier quoted context omitted.

It’s still all Amazon, the single publicly traded company. Legal shenanigans/optimizations don’t change that. The other commenter was referring to AWS the org over Amazon Retail or Devices (other orgs).

Context: I worked at Amazon Retail for 10 years. Amazon Retail and AWS are the same legal entity for stocks, but other than that they might as well be separate companies. Retail uses AWS with all the same APIs and quirks as any other company. The only thing different is the negotiation on price (which many large companies also do). Meanwhile, AWS is apathetic towards feature requests from Retail, and especially opera…

GECKO all the way. :) I think AWS gave a reasonable price to Retail. The migration caused the biggest outage of the website but at the end there was some pretty nice cost saving on the YOY infra cost.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#296

Post by adql » Thu, May 04, 2023, 10:09 AM UTC

tough wrote: post #267

kerkeslager wrote: post #242

Earlier quoted context omitted.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution. > Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employe…

What if you have a really big database? Still 2minutes?

Kinda depends if you can run replication from aurora to regular database

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#297

Post by adql » Thu, May 04, 2023, 10:10 AM UTC

onion2k wrote: post #165

abrookewood wrote: post #73

Earlier quoted context omitted.

Yep, expect the Lambda team to raise hell.

Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available? I doubt that. They probably understand that Lambda is a good service for some things and not for others, and using it as a part of deploying things to AWS is a great idea but using it where it doesn't fit makes all of AWS look bad (i…

> Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available?

They most definitely want to as that would most likely mean more money (and promotions) is flowing there.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#298

Post by sethammons » Thu, May 04, 2023, 10:10 AM UTC

bberrry wrote: post #162

lastangryman wrote: post #143

Earlier quoted context omitted.

To most people, "mono" refers to a single codebase, not a single deployed instance. I've worked on many monoliths that run multiple instances in production. Microservices are no more or less scalable than a monolith. The main benefit of Microservices is allowing multiple teams to work independently from each other without everyone "stepping on each others toes". You can have scalable monoliths and unscalable microser…

Thank you for clarifying. It's seems my definition doesn't correspond to others'. Then do we lack a word for a monolithic application that handles everything and that can only have one instance running?

I've always heard it as "single instance."

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#299

Post by easton » Thu, May 04, 2023, 10:10 AM UTC

datadeft wrote: post #280

bootsmann wrote: post #241

Earlier quoted context omitted.

Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)

It is possible[1]. You are better off this way than the AWS "native" way. 1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...

The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#300

Post by adql » Thu, May 04, 2023, 10:10 AM UTC

__turbobrew__ wrote: post #140

YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

>Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick.

You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for