Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

311–320 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#311

Post by selcuka »

bobsmooth wrote: post #248
sgtnoodle wrote: post #89

Earlier quoted context omitted.

Over 15 years ago now, I was an intern at Toyota. We were working with an in-house python based framework for doing cool/terrible drive-by-wire things with test cars. I had a project to work around a bottleneck of the framework. It could only process about 70 CAN frames per second before running out of CPU. The vehicle's CAN bus had several thousand per second, though. At the time I was able to fix the problem by add…

>They converted every 8-byte payload buffer into a canonical binary representation, i.e. ascii strings of 1's and 0's. Honestly sounds like something I'd do but I've never programmed anything more dangerous than a toaster let alone a car.

You would probably do a little bit of research after seeing the performance of your code. It's one thing to code the prototype sloppily, it's another to push it to prod.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#312

Post by ivanche »

nodefortytwo wrote: post #272
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

To be fair to AWS, they do work really hard to (at least at an account level) to optimize workloads with you. They do this so overall you'll move more workloads to them. its quite simple, if workload x can be done 100% cheaper on-prem then its an obvious move (probably) if AWS manage to get that closer to 30-40% then the operational benefits of using AWS make more sense, more workloads, more total spend.

You mean 50% cheaper on-prem, right? 100% cheaper is 0.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#313

Post by withinboredom »

jmillikin wrote: post #286
withinboredom wrote: post #180

Earlier quoted context omitted.

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

1. Add a test with a time-bomb (such as a test certificate with 365-day duration), wait a year, and now your test fails without having changed. 2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing. 3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing. 4…

Hmm. We never had those specific issues. For (1), we had time bombs for sure, but those usually highlighted coverage issues. Dev ops would disable your test and tell you to fix your code.

IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests.

For (3), all dependencies were committed as part of the repo. All dependencies had to be reviewed for any issues before being used, so this made sense. You simply weren’t allowed to just randomly include a new dependency without a review/PR to add it.

For (4), our dev environments had less memory than build machines and the same as production. If you couldn’t build it in a dev environment, it wasn’t getting committed without special treatment from dev ops (and a really good reason).

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#314

Post by pauby »

lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

It's already on the Wayback Machine https://web.archive.org/web/20230504060528/https://www.prime...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#315

Post by withinboredom »

Akronymus wrote: post #291
anonzzzies wrote: post #269

Earlier quoted context omitted.

Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.

Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?

Don’t merge. Rebase only. Keep a linear history.

When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc.

If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#316

Post by boredumb »

abluecloud wrote: post #309
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

It's honestly like a cult and a desire to want to "do it right" on AWS. The last few projects I've spent so much time setting up code deploy, load balancers, certificates, SES, route 53... This newest project, I've gone to heroku with everything being basically a few clicks to get setup.

It seems like another case of the road to hell being paved with good intentions, most places want/need redundancy and some managed devops and so a few ec2 instances and managed RDS is affordable enough and checks a lot of boxes, but after people start down this path it seems almost irresistible to start drilling down into managed kubernetes, spark jobs, to start ingesting some events we'll just introduce glue, and that plugs right into S3, and look how easy it is to plug in athena, add some quick alerting with cloudwatch, and the next thing you know you're vendor locked and having to hire a full time devops person with AWS experience to configure, manage and keep on top of it all.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#317

Post by adql »

NBJack wrote: post #81
freeone3000 wrote: post #61

Earlier quoted context omitted.

Your asks seem easily answered with docker + kubernetes. Actually, this is in fact the use case for kubernetes — a fault tolerant distributed system running arbitrarily, simply packaged code. This has to be what you’ve tried — what issue are you running into?

Kubernetes isn't something I'd put in the same sentence as 'easy'. Docker is a close contender for the same. I still recall the day when my local Docker builds necessitated a new router to properly manage streaming traffic at home while I downloaded a few GBs of layer images. Or the time I wanted to setup a 'simple' hosted Kubernetes cluster of my own in my lab for testing, only to discover the nightmare that is netw…

When they say k8s is "easy" they mean "for developers", not people/automation running it.

Like, compared to implementing hitless rollback over bare metal services k8s way is "easy", just set some stuff in YAML and have proper healtchecks in your app.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#318

Post by alpos »

"We built a video stream processor by splitting every 1080p+, multi hour long, 30-60fps video into individual images and copying them across networks multiple times."

Not surprising that didn't go will. This strikes me as a punching bag example.

Anyone who has worked with images, video, 3d models, or even just really large blocks of text or numbers before (any kind of actually "big data") knows how much work goes into NOT copying the frames/files around unnecessarily, even in memory. Copying them across network is just a completely naive first pass at implementing something like this.

Video processing is very definitely a job you want to bring the functions to the data for. That is why graphics card APIs are built the way they are. You don't see OpenGL offering a ton of functions to copy the framebuffers into ram so you can work on them there only to copy them back to the video card. And if you did do that, you will quickly find out that you can be 10x to 100x more efficient by just learning compute shaders or OpenCL.

You could do this in a distributed fashion though, but it would have to look more like Hadoop jobs. I predict the final answer here, if they want to be reasonably fast as well, is going to be sending the videos to G4 instances and switching the detectors over to a shader language.

In general, if the data is much bigger than the code in bytes, move the code, not the data.

IO is almost always the most expensive part of any data processing job. If you're going to do highly scalable data processing, you need to be measuring how much time you spend on IO versus actually running your processing job, per record. That will make it dead obvious where you should spend your optimization efforts.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#319

Post by Guid_NewGuid »

boxed wrote: post #112
The_Colonel wrote: post #52

Earlier quoted context omitted.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

I built a little microservice on the side of my monolith for PDF creation. It used headless chrome and ghostscript to render html to a nice PDF. The problem I had with having that code inside the monolith was that it increased my docker image creation for deploys by a lot . And that code pretty much never changed anyway. I did feel a bit embarrassed having to make a microservice after having argued against them so mu…

I feel like it's a semantics thing. To me the meaning of microservices has undergone semantic drift to the antipattern in the article where every little component or database table is its own service with an associated "pizza team".

It's fine to just have plain "services" to do things like this where you need to leverage another OS/framework/whatever and just hive off something like PDF conversion while your core application remains a monolith.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#320

Post by datadeft »

adql wrote: post #305
datadeft wrote: post #287

Earlier quoted context omitted.

Microservices make sense for a lot more than 5%. If fact I think it is much closer to the 80/20, 80% working on serverless, 20% not. Video streaming obviously not going to work on AWS Lambda to begin with.

They really don't unless you routinely put a bunch of tiny self-contained apps. The "pain" of figuring out how to deploy your "normal app" quickly amortizes over just how much easier and more reliable code is.

I finally realised after using Lambda for almost a decade (started to use it when it was released 9 years ago) that instead of think about apps that you map to lambda functions you should think about features instead.

A simple example: I have a SPA that has the following features: auth(login, logout), dashboard, feature a, feature b. I can write a few very simple lambda functions and deploy these the same way (IaC). What do we (my team) win? We can implement each function in a language we want. You have a feature that is too slow? Rewrite it in Rust. You have an amazing Python lib for feature a? Use Python. What else? We almost never touch auth, so if a feature has a bug it does not impact the entire application. Security is better because we can allow individual functions to access part of the infra they really need to access. Lambda functions can call other lambda functions as well.

Downside is that we cannot use a shared cache that is easy with a monolith. People need to design the boxes well which functionality goes to which lambda function. We have to use distributed trace ids to track requests.

Post reply on HN