Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

421–430 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#421

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

As an exaws senior dude we never looked at our service stack as a sell at any cost, but as a continuum of service offerings that could be assembled to be more cost optimal at higher operational burden to (mostly) ops free at a higher premium. The goal was to provide a lego kit of power tools and disappear from view tools. At least in my org we never tried to upsell or convince customers of architectures that accreted revenues at their expense, we tried to honestly assess their sophistication and desire for ops burden and complexity vs cost savings by building it themselves with the lower level kit. By our measure using aws brought us business, and we were generally more motivated by customer obsession over soaking them. I know Andy definitely had that view and drilled it into our collective heads. In many ways as an engineering minded person I appreciated the sentiment as I enjoy solving problems more than screwing people out of their money for sport.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#422

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

I’d note each of what you mentioned cost $0 at zero scale and nominal $ at small scale. But you’re right, engineers new to aws try to flex all the kit together for not much benefit. For a semi static website all you need is s3+cloud front+api gateway+lambda+dynomdb for state. This would cost you basically $0 for small scale, and there would be nothing to monitor. It either works or aws is down.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#423
post #274

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed". Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. My candid reaction: Seriously? WTF? I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and…

I wouldn't be surprised if the actual story underneath was that they got to a "works well enough" implementation and then forgot about the inefficiencies until someone looked at costs, connected the dots, and went "ok yeah we need to optimize this architecture."

I've seen some staggering cost savings realized because someone happened to notice that an inefficient implementation that wasn't a problem two years ago at the scale it was running at back then did not age well to the 10x volume it was handling two years later. The reason it hadn't fallen over was that horizontal scaling features built into the cloud products were able to keep it running with minimal attention from the SRE's.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#424
post #278

Earlier quoted context omitted.

Seems a bit of a dark pattern to have shortcut to onboard and not offer the same shortcut to offboard. Just like easy subscribe-online publications that will have you call during 2 hours with a rep pushing you discounts or whatever to cancel such subscription. Just not cool

I'd characterize this asymmetry (convenient shortcut IN, standard export OUT) as a predictable, transparent, minor annoyance -- not a "dark pattern" representing deceptive or unethical practices.

OK. yeah maybe dark pattern was a too strong term for just the annoyance...

Just I'd try to stay away from amazon if I could

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#425

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

I don’t really agree that this somehow exposes those tools as bad. It more shows that they weren’t that well suited for this particular use case.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#426
post #292

Earlier quoted context omitted.

I'm intentionally ignoring any of the sarcasm in your comment. The time needed for a db dump is always dependent upon the amount of data. This is true regardless of the db software or where it's running.

I can't think of any project I've worked on where the main data base could be backed up and restored to a different database in "2 minutes" The sarcasm was warranted.

Isn’t the suggestion that the “active work” is very little and most of the time you’re just waiting?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#427

Earlier quoted context omitted.

Doesn't Docker use cached image layers to solve that? Your PDF rendering could be in one layer that never changes, and the rest goes on top.

The problem is when you have more than one such service. Now when one of them changes, all of them need to be rebuilt. You can solve this with multi-stage builds, but those only work if your build result can be easily copied.

Is the issue here that images shouldn't be thought of as layers, but rather a tree of cached directory nodes? I don't quite follow what's meant by building here, are you referring to compiling or merging the resulting build artifacts into a final container image?

Non-copyable build outputs sound a bit wild - you're thinking of builds that encode absolute paths into the output binaries?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#428

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

I kind of see the opposite. Relying heavily on stuff like lambda has scaling limitations but it’s fast to get up and running. Built-in interactions between AWS services can do a lot of the lifting for you. And then if you find out that’s not a great fit for what you’re doing you can put in more bespoke pieces.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#429
post #268

Breaking things into tiny functions and putting them on many different servers incurs tradeoff costs in both complexity and compute. There is a complexity cost in having to deal with the setup, security, and orchestration of those functions, and a compute cost because if the overall system is running constantly it will be less efficient and therefore more expensive than running on one box.

I agree on the tradeoffs you have to make. The main cost driver here was storage and traffic, though.

Good point. "communication" should also be on the list. I don't think storage is technically the tradeoff in this case even though it's S3. It's the traffic between those components that's costing them.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#430
They basically underestimated the cost of moving millions of small files to and from S3; it kinda makes sense if they want to save those images for a long time, but in this case it was for semi-real-time error detection, which is much faster to do in-memory.
Post reply on HN