Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

481–490 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#481

Post by baobabKoodaa »

jon-wood wrote: post #222
lucideer wrote: post #157

Earlier quoted context omitted.

Have you had AWS tech advisers advise teams in your company to go with this stack? Because I haven't. AWS doesn't have an equally distributed interest in selling all of its products. Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon: the first type of products are great for customer acquisition, the role of their sales folk is…

I've never knowingly had an AWS solutions architect recommend something because it would make AWS more money in the short term. The most frequent advice I've seen from them has been on how to give AWS less money by making use of different features, or changing how particular services are being used.

You sound very confident in your estimation of other peoples' motivations and skills. Do you imagine AWS solution architects coming to you directly with "you should pay for this service because it makes more money for AWS"?

I've done the AWS solutions architect associate level cert and I can tell you first-hand experience that in order to pass the exam you need to memorize a lot of AWS propaganda that was written primarily to optimize AWS profit, not to optimize customer satisfaction. How many of those solution architects take those materials with a grain of salt vs how many of them genuinely believe that crap, I don't know.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#482

Post by baobabKoodaa »

steveBK123 wrote: post #337
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

Yes, and it attracts just the wrong kind of dev/architects. At a previous shop, we hired a cloud architect to drive our "cloud adoption". He of course bet the farm on a set of new AWS services that were barely in version v0.9 to be the backbone of the system he architected. It quickly became clear even he had no experience with the set of tools & services he had advocated, and the whole thing went off the rails slowl…

But it's fun to play around with new toys!

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#483

Post by baobabKoodaa »

pjc50 wrote: post #384
henry_bone wrote: post #86

Earlier quoted context omitted.

Seems like Conway's law applies:- "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." -- Melvin E. Conway https://en.wikipedia.org/wiki/Conway%27s_law

Came here to say this - and it applies in the other direction. Microservices allow you to split work between teams without having to coordinate deployment and iteration cadence quite so tightly. If you have a single team, you shouldn't be doing microservices.

I mostly see organizations with multiple internal dev teams who all have shared responsibility over all of the microservices (e.g. no-one is responsible for any service). The worst of both worlds: all of the complexity of microservices architecture, without the benefit of specialization and splitting work between teams.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#484

Post by goodpoint »

motbus3 wrote: post #251
goodpoint wrote: post #245

Earlier quoted context omitted.

[flagged]

I do not care about the downvoting it is unfortunate that there is no comment back saying why the person disagrees

That's my point. Silent downvotes.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#485

Post by latexr »

oefrha wrote: post #261
latexr wrote: post #216

Earlier quoted context omitted.

Do you have a link to documentation on that method of highlighting text? Hard to search for due to the all the non-letter characters.

Text fragments: https://wicg.github.io/scroll-to-text-fragment/ Currently supported on all non-Firefox major browsers. https://caniuse.com/url-scroll-to-text-fragment

Thank you.

Seems Brave is an exception amongst Chromium browsers. They don’t implement it for privacy reasons.

https://github.com/brave/brave-browser/issues/22906

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#486

Post by Paul-Craft »

kerkeslager wrote: post #474
Paul-Craft wrote: post #468

Earlier quoted context omitted.

I think you've failed to consider how impractical even doing a SQL dump can be on a large-ish database, forget about the reloading time.

I haven't failed to consider that, you've just failed to read where I explicitly mentioned that. Consider: is that impracticality caused by Amazon creating vendor lock-in? Or is that impracticality caused by the fact that reading terabytes of data from storage, transferring it over the network, and writing it into storage is inherently slow because of the physical limitations of hardware, no matter what vendor you're…

I didn't miss where you mentioned it. I missed where you considered it. As in, you know, gave any thought to the practicalities involved.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#487

Post by kerkeslager »

Paul-Craft wrote: post #486
kerkeslager wrote: post #474

Earlier quoted context omitted.

I haven't failed to consider that, you've just failed to read where I explicitly mentioned that. Consider: is that impracticality caused by Amazon creating vendor lock-in? Or is that impracticality caused by the fact that reading terabytes of data from storage, transferring it over the network, and writing it into storage is inherently slow because of the physical limitations of hardware, no matter what vendor you're…

I didn't miss where you mentioned it. I missed where you considered it. As in, you know, gave any thought to the practicalities involved.

You're really claiming you can read my mind to know what I have and have not considered right now, despite me bringing it up specifically. Alrighty then.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#488

Post by manv1 »

alpos wrote: post #318

"We built a video stream processor by splitting every 1080p+, multi hour long, 30-60fps video into individual images and copying them across networks multiple times." Not surprising that didn't go will. This strikes me as a punching bag example. Anyone who has worked with images, video, 3d models, or even just really large blocks of text or numbers before (any kind of actually "big data") knows how much work goes int…

I think the realtime requirement removes hadoop as an option. They might have considered using HDFS as the data store instead of S3, since putting lots of objects into s3 is expensive. Or just using a big EFS volume instead of S3.

It would be nice to know how much latency there was in the microservice version vs the monolithic version.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#489

Post by Ensorceled »

kerkeslager wrote: post #453
Ensorceled wrote: post #339

Earlier quoted context omitted.

I can't think of any project I've worked on where the main data base could be backed up and restored to a different database in "2 minutes" The sarcasm was warranted.

If only you had read the whole sentence, you might have saved yourself a bit of righteous anger.

You seem to be the one angry here.

I'm not sure what I was supposed to take away from your cryptic sentence. Is there a two minute solution to this problem that you are smugly keeping to yourself so you can mock people replying to you?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#490

Post by zenbowman »

Shipping around individual video frames between components is really an astonishingly bad idea.

Microservices seem to be a decent idea with a terrible name. The idea of running services that are small enough that they can be managed by a single team makes sense - it enables each team to deploy their own stuff.

But if you break things down further, where you need multiple "services" to perform a single task, and you have a single team managing multiple services - all you do is increase operational & computational overhead.

Post reply on HN