Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

471–480 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#471
I'm happy to see -- in the discussion here-- the continued backlash against microservices and the deleterious effects it has had on software complexity, and data modelling.

But I think it's interesting that if we took a time machine back to 2014 or 2015 the tone here would be quite different, and microservices were all the rage on this forum as I recall.

I like to hope that the industry learns from its failed trends, but I'm now old enough to see this is rarely the case.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#473
post #256

Earlier quoted context omitted.

Having autonomous teams is great for scaling and allowing everyone to go fast, without teams constantly blocking each other. Having hundreds of engineers work in a single monolith in a single repo without any kind of (enforced) boundaries is a one way ticket to a big ball of mud. You need to invest heavily in tooling to make it work, and e.g. Google does so. Having a network in between teams is a relatively easy way…

It allows everyone to go fast as long as the work is constrained within one service. It goes very slow once service / team coordination needs to happen and one team alone is not able to deliver the feature. This then often leads to services duplicating logic, amassing responsibilities in order to do as much as possible within "my" service to avoid this coordination bottleneck.

Then you're either not setting up your team responsibilities right, or you're not allowing cross-team contribution, both are fixable mistakes.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#474

Earlier quoted context omitted.

If only you had read the whole sentence, you might have saved yourself a bit of righteous anger.

I think you've failed to consider how impractical even doing a SQL dump can be on a large-ish database, forget about the reloading time.

I haven't failed to consider that, you've just failed to read where I explicitly mentioned that.

Consider: is that impracticality caused by Amazon creating vendor lock-in? Or is that impracticality caused by the fact that reading terabytes of data from storage, transferring it over the network, and writing it into storage is inherently slow because of the physical limitations of hardware, no matter what vendor you're using?

It's a bit odd for me to be in the position of defending Amazon here. I genuinely don't like them, don't use them, and generally do think they're guilty of creating a lot of vendor lock-in. But this is legitimately not an example of any of that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#475
post #464
post #409

Earlier quoted context omitted.

I counter your anecdote with mine! A cloud vendor (who will be nameless as I signed an NDA specifically that prevents me from disparaging them; but one of the big three) ran out of capacity for me and it was 3 months before they managed to fix it. -- that was with a couple million a month in spend. Cloud is still servers; you just depend on someone elses capacity management skills and you hope that there isn't a rush…

3 months vs 2 years, I'll take it :)

I think the main point here is that there is no limit to incompetence. And sure, having your own servers allow for some goofs that won't happen with cloud (the opposite is also true). But your org had the means to fix the issue, and they choose not to. That has fundamentally got nothing to do with technology choice.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#476
post #261
post #216

Earlier quoted context omitted.

Do you have a link to documentation on that method of highlighting text? Hard to search for due to the all the non-letter characters.

Text fragments: https://wicg.github.io/scroll-to-text-fragment/ Currently supported on all non-Firefox major browsers. https://caniuse.com/url-scroll-to-text-fragment

[deleted]

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#477

Earlier quoted context omitted.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

This can be done with DMS. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introd...

Huh. I suppose you can. The pricing looks a little opaque though, and the fact that it took 3 hours after I wrote my comment for yours to show up kind of implies it's a bit of an obscure service.

I will also mention that the AWS team we were working with on this didn't mention DMS, and, when directly asked, literally told me there was no easy way to do an Aurora -> RDS migration.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#478
post #299

Earlier quoted context omitted.

It is possible[1]. You are better off this way than the AWS "native" way. 1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...

The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.

We've actually observed the opposite at our company. Moving from a Python 3.8 built-in to Docker based changed our response times from about 40ms to 30ms on average.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#479

Earlier quoted context omitted.

Isn’t the suggestion that the “active work” is very little and most of the time you’re just waiting?

Well, the elapsed time is how long I need production to be offline so there are a few more people "just waiting"

Why would you take production offline to swap databases? Elapsed time is how long you need to run the new database alongside the old one.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#480
post #200

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

But they migrated to AWS ECS that still is an expensive serverless AWS stuff, just fully managed by Amazon.

This is simply incorrect. ECS doesn't cost anything other than what you're paying for the EC2 instances that you place your tasks on. Fargate does, but that's not what they're using.
Post reply on HN