So one team at Prime for one specific application learned, that serverless was not the ideal compute model for their workload. Wow.
Scaling up the Prime Video audio/video monitoring service and reducing costs
521–526 of 526 posts
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#522Earlier quoted context omitted.
> Microservices are no more or less scalable than a monolith. This is not fully true. A microservice architecture is more finely scalable than a monolith. To take a very basic example, if you have a peak of users watching a video you can scale up the microservice dedicated to serving videos, but not scale up the service dedicated to users signups, which isn't having an increased load.
> A microservice architecture is more finely scalable than a monolith. Apologies, but I strongly disagree and I'm going to go on a bit of a rant here.... This is a myth, and one of the reasons people are making these ridiculous architecture descisions. If you have a monolith that serves videos and enables signups, you can deploy as many instances of that as you like based on the highest need. It doesn't matter if use…
> Microservices are nothing to do with scalability. They are about how you organise code and teams to achieve better development velocity.
I don't think this is strictly true; even though microservices are usually used that way.
Scaling up everything even when not needed has it difficulties. You can have lots of unnecessary initialisation tasks, lots of unused caches warmed up, database and socket connections that are not needed, complexities in work sharing algorithms etc.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#523Earlier quoted context omitted.
This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al. A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.
I just spent the better part of two years advocating, pushing, and fighting for months to add new bandwidth to our datacenter. Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other. While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a mi…
Or maybe there's a mid point. It's not datacenter or cloud. There are providers offering physical servers for rent for example. Lots of combinations in-between.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#524Earlier quoted context omitted.
The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.
I used to believe the same thing, I've began thinking it might now be false yet often repeated. AWS claims that: > Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives.[0] This[1] article shows almost no discernable difference in .NET cold start times between containerised and regular lambdas. It's easy to imagine developers pushing up b…
The situation does keep changing - AWS does optimize things.
I'm not so sure it's a black/white true/false. Depends on what goes in the docker image. It's something like for larger deployments docker is faster but for small deployments it's the other way.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#525Earlier quoted context omitted.
This can be done with DMS. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introd...
Huh. I suppose you can. The pricing looks a little opaque though, and the fact that it took 3 hours after I wrote my comment for yours to show up kind of implies it's a bit of an obscure service. I will also mention that the AWS team we were working with on this didn't mention DMS, and, when directly asked, literally told me there was no easy way to do an Aurora -> RDS migration.
Which is weird. This has been around for a few years, and, at least I thought, was a fairly popular service.
Previously it wasn't great, but since they started using Logical Replication for Postgres, it's gotten far better than it was in the past.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#526The headline is a bit of a misnomer. This happens in large businesses all the time (which isn't to say it's "good", hardly is, but it suggests the causation is incorrect here, which then indicates the conclusion is entirely off-base):
1) We have sexy new product! Everyone use it so we have some use-case stories to tell and we look credible! Who cares if it's not the right tool for the job! We need a splashy way to use hackneyed business speak like "we're eating our own dog food" at the next user con so all the IT middle managers there will fight over early access and adoption. PROFIT! (Screams of technology teams in the background of "a knife is the most expensive, useless pry tool you can buy, but whatever, you are not listening, mmmkay").
2) A few quarters/years later (if you're lucky and you made it or someone with enough gravity in their title finally saw the light): Why is expense so high in this business unit? This is insane! Let's go back to a more sane architecture. (Screams of technology teams going back to what was working in the first place, but was not sexy nor necessarily new now that no one is watching and hype cycle is over)
Does this mean that serverless is useless? Dumb? Uneconomical? No way. For bursty, very short running workloads, it can be GREAT and INCREDIBLY economical.
What is useless and "dumb" is whomever thought that Prime Video's encoding workloads were going to do anything but increase cost and were somehow a fit for a system whose business case specifically necessitates bursty, shorter workloads that are primarily scale-to-zero for significant periods of the day/week/month.
It was a marketing stunt gone horribly wrong: intentional or not, but that doesn't repudiate the value of "serverless" for the right workloads, it just proves you better really understand the technology and the business case and the scale economics, and that goes for any technology.