Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

411–420 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#411

Post by whstl »

withinboredom wrote: post #315
Akronymus wrote: post #291

Earlier quoted context omitted.

Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?

Don’t merge. Rebase only. Keep a linear history. When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc. If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relyi…

"Don’t merge. Rebase only. Keep a linear history."

This. It makes life so much simpler. With teams that don't have a lot of experience with git, however, I tend to use the "Squash and Merge" feature, coupled with forcing a linear history.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#412

Post by adql »

datadeft wrote: post #320
adql wrote: post #305

Earlier quoted context omitted.

They really don't unless you routinely put a bunch of tiny self-contained apps. The "pain" of figuring out how to deploy your "normal app" quickly amortizes over just how much easier and more reliable code is.

I finally realised after using Lambda for almost a decade (started to use it when it was released 9 years ago) that instead of think about apps that you map to lambda functions you should think about features instead. A simple example: I have a SPA that has the following features: auth(login, logout), dashboard, feature a, feature b. I can write a few very simple lambda functions and deploy these the same way (IaC).…

I kinda thought about making "monolithic lambda", where there is just interface to get the request, respond to request, logging, and maybe some queue to talk with other components, for some of my personal tools.

Basically cut down the cruft when deploying another small self-contained feature but still keep the code running (savings of few MB memory are meanigless if you just have few dozen features that might run at the same time anyway).

Then I realized it's basically reinventing the ancient idea of "application server" like JBoss and EJB... which is kinda the case for lambda anyway.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#413

Post by lxgr »

YetAnotherNick wrote: post #114
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

That's my experience too. Microservices are first and foremost a technology to dilute responsibility, and if you're clever about it, you can even let it fall through the cracks completely.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#414

Post by lxgr »

Akronymus wrote: post #291
anonzzzies wrote: post #269

Earlier quoted context omitted.

Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.

Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?

Requiring a passing integration branch before merging to master. Merging the integration branch then becomes a fast-forward merge.

Alternatively, if you have a low enough merge volume, requiring mergers (by policy) to squash and rebase (and re-run tests before attempting to merge) can work too, as others have already mentioned.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#415

Post by lxgr »

arp242 wrote: post #209
kubanczyk wrote: post #198

Earlier quoted context omitted.

Two green changes merged to a green main can produce a red main. This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default: > Require branches to be up to date before merging > Whether pull requests targeting a matching branch must be tested with the latest code.

> Two green changes merged to a green main can produce a red main. I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.

It completely depends on your organization's merge volume and your codebase's complexity. I've seen it happen many times working on a large monolith. But requiring a passing integration branch build before merging to master, or a merge queue, solves that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#416

Post by adql »

moonchrome wrote: post #391
adql wrote: post #366

Earlier quoted context omitted.

well, they have no incentive to not make you pay for CPU time on your application startup

The problem isn't paying for cold start - the problem is they make the low ram lambdas very very niche by CPU scaling - you can have a 256 mb web server that talks to a database easily - and that's their supposed selling point - but having it served on ~300MHz CPU is really really limiting - and they should be upfront about that. If you went to a car rental and they told you we have a cheap car that's slower when you…

> and they should be upfront about that.

Do you need a screenshot and red box around the text or would you believe me if I tell you it is written on their lambda pricing page near the beginning ? It's also written in docs about configuring lambad functions so at this point it is PEBKAC/RTFM issue, not "them not being upfront"

And frankly it is done that way because they have standarized machines, scheduling CPU heavy/memory light and cpu light/memory heavy is extra complexity. I mean ,they should, but they have no real incentive to, as in most cases apps written in slower languages are also memory-fatter so it fits well enough

> If you went to a car rental and they told you we have a cheap car that's slower when you add passengers - and then you drive it to pick up your wife and it turns out it only goes 20 km/h when your wife gets in - you would be rightfully mad.

Getting lowest tier one is more like renting a 125cc bike than a car if anything. You can do plenty with that limit in efficient language too.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#417

Post by lxgr »

nomilk wrote: post #195
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

I haven't worked on a microservice architecture yet, but this is a very interesting idea that I hadn't heard before. That micro services can potentially give greater visibility of each team's performance, improving accountability.

It's incredibly easy to game that architecture in a low-trust environment, though. If a team owns the interface definition of their microservice, they can just declare all callers' problems an instance of "holding it wrong".

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#418

Post by moonchrome »

adql wrote: post #416
moonchrome wrote: post #391

Earlier quoted context omitted.

The problem isn't paying for cold start - the problem is they make the low ram lambdas very very niche by CPU scaling - you can have a 256 mb web server that talks to a database easily - and that's their supposed selling point - but having it served on ~300MHz CPU is really really limiting - and they should be upfront about that. If you went to a car rental and they told you we have a cheap car that's slower when you…

> and they should be upfront about that. Do you need a screenshot and red box around the text or would you believe me if I tell you it is written on their lambda pricing page near the beginning ? It's also written in docs about configuring lambad functions so at this point it is PEBKAC/RTFM issue, not "them not being upfront" And frankly it is done that way because they have standarized machines, scheduling CPU heavy…

>Do you need a screenshot and red box around the text or would you believe me if I tell you it is written on their lambda pricing page near the beginning ? It's also written in docs about configuring lambad functions so at this point it is PEBKAC/RTFM issue, not "them not being upfront"

Simple CPU time calculator on the pricing calculator page when you enter the RAM would be sufficient, linking to the said docs. Trivial to implement, really cleans up things when planning resource costs.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#420

Post by runlaszlorun »

adql wrote: post #402
jabradoodle wrote: post #359

Earlier quoted context omitted.

It's not really the same spot in that your paying monthly rather than upfront. Devs tend to think about total $, the business/accountants do care about Opex vs Capex. Also it's going to be simpler to provision your base (commited use) on the cloud and then handle bursts on the cloud, than it is to have your base on prem and burst to the cloud.

> It's not really the same spot in that your paying monthly rather than upfront. Devs tend to think about total $, the business/accountants do care about Opex vs Capex. You can buy physical servers in leasing ,turning it into opex You can also rent them for little bit extra via managed dedicated servers from vendors like OVH.

I think this point isn’t made often enough.

Not going with a big cloud provider def doesn’t mean that you need to buy physical servers and build an on-prem data center.

Post reply on HN