I'm waiting for the AWS Lambda team to talk to Marketing to get this taken down ...
Scaling up the Prime Video audio/video monitoring service and reducing costs
71–80 of 526 posts
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#72Feels more like the initial version was a prototype not meant to scale
Wouldn’t have expected prime to be pushing around images on s3
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#73My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…
Yep, expect the Lambda team to raise hell.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#74My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#75I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.
I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…
[dead]
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#76Sending video frames between services is expensive, also doing per state transition hosting on things doing state transitions multiple times per second in a single stream is also expensive... Like, did they even think about cost when designing this the first time?
why should they, they're richer than God!
You don't get (and stay rich) by wasting all your resources.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#77Am I right in understanding this is just their defect-detection system?
Yes, this is just the defect detector and not the actual video streaming service.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#78Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.
> AWS Step Functions charges users per state transition
Apparently they didn’t know about the EXPRESS execution model, or the much improved Map state. The story seems to be one of failing to do the math and design for constraints rather than an indictment of serverless.
I have to agree with others - it is amazing this article saw the light of day.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#79I've never seen successful micro services if the starting point is not a monolith. The most successful ones I've seen are hybrid ones where some parts needed to be scaled are refactored as a micro service to run in parallel.
Bang on. A friend I work with used to say "microservices are for scaling teams, not tech" which I liked.
Even with monolith -> microservices I've seen it go wrong. One Go application I worked on it would take a senior engineer a week to add a basic CRUD endpoint as the code had been split in to microservices along the wrong boundaries. There was a ridiculous amount of wiring up and service to service calls that needed done. I remember suggesting a monolith might be more appropriate, and was told it used to be a monolith but had been "refactored to microservices"...
This type of stuff can literally kill early stage companies.
Re: Scaling up the Prime Video audio/video monitoring service and reducing costs
#80Earlier quoted context omitted.
And then team 1 needs to upgrade pandas to 2.0, but team 2 is still on pandas 1, so when the main app pulls them in nothing works, so you need to start a cross-team committee to schedule the work to upgrade a single library... Separate services aren't a silver bullet, but as an fyi to the younger software developers, we tried "just have all teams work on the same code base and deployable artifact" for a long while an…
All the teams will need to migrate sooner or later so figuring out all of the potential problems in migrating and having everyone do it an once is more efficient than each team needing to figure it out separately.
That's not how it plays out in reality. usually nothing gets done because "upgrade this package" is never on anyone's priority list. Or teams end up doing shit like JAR shading or forking and renaming a package with some _v2 or whatever suffix to be able to support both the old and new version simultaneously in the main code base. And then of course nobody ever updates the runtime (hello, enterprise monoliths still running on Java 6/7!) It ends up being a complete mess.