Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

461–470 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#461

Post by cglong »

cmrdporcupine wrote: post #387
cglong wrote: post #135

Earlier quoted context omitted.

My team owns an API monolith that hosts several completely unrelated endpoints. I keep thinking this would be a good candidate for breaking into microservices, but I do wonder if I'm buying into the hype.

When you say unrelated, are you sure? Do they share -- or should they share -- a common underlying relational data model? My biggest grief with microservices is the fact that it's effectively become a war on having a coherent logical normalized relational data model inside an organization.

This is a good question to think about. Right now, they don't have any relationship beyond serving different parts of my group, but perhaps they could be redesigned in a more cohesive way.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#462

Post by cglong »

bottlepalm wrote: post #168
cglong wrote: post #135

Earlier quoted context omitted.

My team owns an API monolith that hosts several completely unrelated endpoints. I keep thinking this would be a good candidate for breaking into microservices, but I do wonder if I'm buying into the hype.

You are. People like you nuked all productivity we had at my last company.

Well, I've been sitting on this thought for two years now, so they probably weren't exactly like me :)

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#464

Post by hobs »

dijit wrote: post #409
hobs wrote: post #397

Earlier quoted context omitted.

I just spent the better part of two years advocating, pushing, and fighting for months to add new bandwidth to our datacenter. Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other. While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a mi…

I counter your anecdote with mine! A cloud vendor (who will be nameless as I signed an NDA specifically that prevents me from disparaging them; but one of the big three) ran out of capacity for me and it was 3 months before they managed to fix it. -- that was with a couple million a month in spend. Cloud is still servers; you just depend on someone elses capacity management skills and you hope that there isn't a rush…

3 months vs 2 years, I'll take it :)

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#465

Post by awaythrow98765 »

Jochim wrote: post #457
steveBK123 wrote: post #327

Earlier quoted context omitted.

Yup, and you are paying the premium of cloud forever, which over some vanilla compute & storage can be a lot. And cloud proponents pretend data center / rack space / server leasing doesn't exist either, for those trying to avoid large up front costs.

I'm a cloud proponent because it means not having to sit through hours of meetings to deploy a $5/mo virtual machine. It also means some poor fuck at AWS gets woken up in the middle of the night instead of me when things go to shit. It absolutely comes at a cost, and might not be the right fit for an organisation that's absolutely on top of it's hardware requirements and can afford to divert resources from new develo…

> I'm a cloud proponent because it means not having to sit through hours of meetings to deploy a $5/mo virtual machine.

And endless orgies of "call for pricing" with hardware vendors and hosting. Shitty websites where you can buy preconfigured servers somewhat cheaply, or vendor websites where you can configure everything but overpay. Useless sales-droids trying to "value-add" stuff on top.

Cloud buys are a lot friendlier, because you only have the one cloud vendor to worry about. Entry level you just pay list price by clicking a button. If you buy a lot, you are big enough to have your own business people to hammer out a rebate on list price, still very easy, still very simple. But overall still more expensive unfortunately.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#466

Post by __turbobrew__ »

adql wrote: post #300
__turbobrew__ wrote: post #140

Earlier quoted context omitted.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

>Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for

Tests can pass before they are merged but fail after merge. Tests can also regress over time and become flaky.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#467

Post by __turbobrew__ »

nikita2206 wrote: post #240
__turbobrew__ wrote: post #140

Earlier quoted context omitted.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

IMO for these rare occasions it should be okay to unmerge PRs. Or what could also be done is you only deploy cherry-picked releases, granted that you have a way of tracking commits in the upstream such that no commit ever gets lost.

That is possible, but at that point you need a integrations/release team to babysit the builds and make sure everything integrates back together cleanly.

The last thing I want to do is have my build broken by someone on another team and then have to track them down and babysitting the revert. That is easily an hour of my time wasted.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#468

Post by Paul-Craft »

kerkeslager wrote: post #453
Ensorceled wrote: post #339

Earlier quoted context omitted.

I can't think of any project I've worked on where the main data base could be backed up and restored to a different database in "2 minutes" The sarcasm was warranted.

If only you had read the whole sentence, you might have saved yourself a bit of righteous anger.

I think you've failed to consider how impractical even doing a SQL dump can be on a large-ish database, forget about the reloading time.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#469

Post by klabb3 »

bottlepalm wrote: post #247
klabb3 wrote: post #226

Earlier quoted context omitted.

Devils advocate, but is it possible microservices need a shared feature-ful message passing layer, and good tooling, to be working well? Eg schema, auth, flow control, ttl, persistence, partitioning etc in the message layer, a la Kafka? I mean it’s kind of implied that microservices can only work if they can talk to each other, and “talking” is a lot more nuanced than we tend to think.

Comm requires massive overhead versus simply 'calling a function'. Calling a function doesn't fail. Maybe the code in it does, but not the call to the function itself. This is why microservice costs often far outweighs the benefits, but they rarely consider the cost in their crusade to 'break up the monolith'

Absolutely. But in the cases where it a monolith is not doable, such as highly heterogenous hardware requirements for different “services”, it seems like the messaging stack plays a more important role (even if that’s only needed in 5% of use cases). Typical HTTP request-response, which is often fine for a monolith, is not enough for building say task queues. A strong messaging layer can reduce the need of ad-hoc wheel reinvention.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#470

Post by cmrdporcupine »

cglong wrote: post #461
cmrdporcupine wrote: post #387

Earlier quoted context omitted.

When you say unrelated, are you sure? Do they share -- or should they share -- a common underlying relational data model? My biggest grief with microservices is the fact that it's effectively become a war on having a coherent logical normalized relational data model inside an organization.

This is a good question to think about. Right now, they don't have any relationship beyond serving different parts of my group, but perhaps they could be redesigned in a more cohesive way.

The core pessimal problem presented by microservices is this: someday some stakeholder is going to ask for some information to be joined together or interlinked -- and that information will have been unwisely put into separate services ... and now you'll be doing joins manually via web service calls -- over the network -- and somewhere the ghost of E.F. Codd is spinning in circles and cursing you.

I think it's actually quite rare for companies to have data so actually autonomous and unrelated that it does not logically relate to anything else in the organization.

Post reply on HN