Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

511–520 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#511

Post by alpos »

manv1 wrote: post #488
alpos wrote: post #318

"We built a video stream processor by splitting every 1080p+, multi hour long, 30-60fps video into individual images and copying them across networks multiple times." Not surprising that didn't go will. This strikes me as a punching bag example. Anyone who has worked with images, video, 3d models, or even just really large blocks of text or numbers before (any kind of actually "big data") knows how much work goes int…

I think the realtime requirement removes hadoop as an option. They might have considered using HDFS as the data store instead of S3, since putting lots of objects into s3 is expensive. Or just using a big EFS volume instead of S3. It would be nice to know how much latency there was in the microservice version vs the monolithic version.

You never get "realtime" in data processing. Actual realtime systems are a totally different animal. Mostly done in the embedded space, the design of a realtime processing system involves setting up fixed time windows for each task that needs compute time and optimizing the code for each task until it fits into the time window for it, on every execution, every time. This is done in order to provide hard guarantees on how fast a system can respond to new data flowing in. It's usually only safety critical systems that actually have such responsiveness and delivery time constraints.

I point this out because how we talk about a problem determines what solutions we even acknowledge as being on the table here. Saying it's a realtime system when it isn't, or thinking we need realtime processing when we don't, makes people throw out solutions per-maturely, that the thrown out solutions are often right answers.

Once you acknowledge that your system will not be "realtime" and you actually don't have the time-boxing and specific time window delivery constraints that actual realtime problem spaces have, you can weigh all of your actual options with an eye for what will be fastest and most efficient given the budget and hardware you have to throw at this problem.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#512

Post by kerkeslager »

Ensorceled wrote: post #489
kerkeslager wrote: post #453

Earlier quoted context omitted.

If only you had read the whole sentence, you might have saved yourself a bit of righteous anger.

You seem to be the one angry here. I'm not sure what I was supposed to take away from your cryptic sentence. Is there a two minute solution to this problem that you are smugly keeping to yourself so you can mock people replying to you?

My guy, read this whole sentence, which has remained unchanged this entire conversation:

"There's no way to move from Aurora to RDS short of... 2 minutes of actual work and a lot of waiting around due to the limitations of the hardware?"

You seem to be having trouble getting past the word "and", so I've helpfully italicized the part you've repeatedly missed or ignored.

Now sure, that's a bit vague, but if you want more details it might have been advisable to ask a question rather than simply ignoring half the sentence because you don't understand it and jumping in with a correction.

And honestly, even if it's vague on some details, there's no universe in which "2 minutes and a lot of waiting around" = "2 minutes". Whatever vagueness you might accuse me of, that fact isn't vague.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#513

Post by Jochim »

jjav wrote: post #502
Jochim wrote: post #457

Earlier quoted context omitted.

I'm a cloud proponent because it means not having to sit through hours of meetings to deploy a $5/mo virtual machine. It also means some poor fuck at AWS gets woken up in the middle of the night instead of me when things go to shit. It absolutely comes at a cost, and might not be the right fit for an organisation that's absolutely on top of it's hardware requirements and can afford to divert resources from new develo…

> I'm a cloud proponent because it means not having to sit through hours of meetings to deploy a $5/mo virtual machine. I'd hope there aren't actually hours of meetings for a single $5/mo VM? But I would hope there are reviews and meetings when deploying enough of these to amount to real money. Companies that don't do that soon enough find themselves with a million dollar AWS bill without understanding what's going o…

> I'd hope there aren't actually hours of meetings for a single $5/mo VM?

Slightly exaggerated in the case of the $5 machine, probably 2-3 manhours total but it took 4 days for it to be deployed instead of ~5 minutes. We did spent tens of hours justifying why the business should spend ~$100 more per month on a production system where the metrics clearly indicated that it was resource constrained.

The same IT department that demanded we justify every penny spent did not apply any of that rigour to their own spending. Control over the deployment of resources was used as a political tool to increase their headcount.

> I would hope there are reviews and meetings when deploying enough of these to amount to real money. Companies that don't do that soon enough find themselves with a million dollar AWS bill without understanding what's going on.

I consider the judicious use of resources to be part of my job as a software engineer. A development team that isn't considering how they can reduce spend, tidy up, or right-size their resources is a massive red flag to me. Organisations frequently shoot themselves in the foot by shifting that responsibility away from the development team. The result is usually factional infighting and more meetings.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#515

Post by knightofmars »

thatwasunusual wrote: post #182
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

The problem I see in many projects, is that they start out as - or implementing - a microservice architecture. I think this is backwards; you should start with a monolith and separate out concerns into microservies if it makes sense, not because it's "cool."

I agree, but aside from it being seen as "cool", what drives some engineers to go microservice first architecture is having experienced the inability of an organization to acknowledge that they actually do need to re-write a monolith as two or more services or undergo a general re-architecting of the monolith. Getting buy-in from the business is extremely difficult as clearly communicating the actual effort and impact that re-architecting the monolith would require is nearly impossible. This is usually due to poor separation of domains via lack of modules within the monolith, spaghetti code, circular or other strange dependency trees, tables with relationships or data that should never have existed in those tables, and a whole other set of other bizarre issues that were due to lack of planning and general discipline by engineers along the way.

If you have a microservice first architecture, the perception is, it's easier to describe effort to re-write an individual service or split it into two services as there is a clearly delineated body of work. Bizarre service-to-service dependencies may still exist and a poorly implemented microservice architecture is still a potential challenge.

Point being, organizations incentivize bad economic decisions on the part of engineers through the inability to recognize that rework is a necessary aspect of developing software and by constantly eschewing rework in favor of feature delivery it sends a strong message to the engineer about what to prioritize.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#516

Post by dbish »

yazaddaruvala wrote: post #230
dbish wrote: post #147

Earlier quoted context omitted.

It’s still all Amazon, the single publicly traded company. Legal shenanigans/optimizations don’t change that. The other commenter was referring to AWS the org over Amazon Retail or Devices (other orgs).

Context: I worked at Amazon Retail for 10 years. Amazon Retail and AWS are the same legal entity for stocks, but other than that they might as well be separate companies. Retail uses AWS with all the same APIs and quirks as any other company. The only thing different is the negotiation on price (which many large companies also do). Meanwhile, AWS is apathetic towards feature requests from Retail, and especially opera…

I worked on both sides. I mostly agree except there are cases of important projects including AWS (like some of the ML work), also the whole aws usage discount/pricing thing is pretty huge and clearly the value in being within the same company. Retail would have a pretty hard time existing nowadays if they weren’t connect to aws imho.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#517

Post by coffeebeqn »

lastangryman wrote: post #326
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

> AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it I was back on AWS for the first time in a few years this week and the amount of new "upsell" prompts in the console is ridiculous. Spin up an RDS instance - "hey, would you like an Elasticache cluster too?". I think AWS are very aware of this behaviour and encourage it.…

I mean that’s upselling 101.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#518

Post by confiq »

kerkeslager wrote: post #512
Ensorceled wrote: post #489

Earlier quoted context omitted.

You seem to be the one angry here. I'm not sure what I was supposed to take away from your cryptic sentence. Is there a two minute solution to this problem that you are smugly keeping to yourself so you can mock people replying to you?

My guy, read this whole sentence, which has remained unchanged this entire conversation: "There's no way to move from Aurora to RDS short of... 2 minutes of actual work and a lot of waiting around due to the limitations of the hardware ?" You seem to be having trouble getting past the word "and", so I've helpfully italicized the part you've repeatedly missed or ignored. Now sure, that's a bit vague, but if you want m…

hey,

that is fair answer! AFAIK, there are two things that consider:

* Aurora do have some vendor-locking feature if I'm not wrong?

* moving from Aurora to PostGres will lead to downtime of 2 minutes + unknown waiting, where this is not a case when you convert from PostGres to Aurora.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#519

Post by confiq »

TexanFeller wrote: post #393
m_mueller wrote: post #208

Earlier quoted context omitted.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Aurora significantly modifies the internals of the DBs, particularly the storage layers. It also makes large changes to how memory is used for Postgres. Query plans can be quite different than with the vanilla version. Once you tune and create indexes based on Aurora's characteristics it's going to be a pain to retune for the unmodified version. Aurora also introduces nasty bugs that don't exist on the RDS version su…

thank you! I needed to read this!
Post reply on HN