Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

391–400 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#391

Post by moonchrome »

adql wrote: post #366
moonchrome wrote: post #271

Earlier quoted context omitted.

Serverless has such bullshit insidious pricing that makes it seem like you're saving money only to figure out you're in shit once you're knee deep in it. For example you'll have to read fine print to find out that 256MB lambda will have the compute power of a 90s desktop PC because compute scales with memory. And to get access to "one core" of compute you have to use like 2GB of memory. Now you may say "serverless is…

well, they have no incentive to not make you pay for CPU time on your application startup

The problem isn't paying for cold start - the problem is they make the low ram lambdas very very niche by CPU scaling - you can have a 256 mb web server that talks to a database easily - and that's their supposed selling point - but having it served on ~300MHz CPU is really really limiting - and they should be upfront about that.

If you went to a car rental and they told you we have a cheap car that's slower when you add passengers - and then you drive it to pick up your wife and it turns out it only goes 20 km/h when your wife gets in - you would be rightfully mad. You could say "why didn't you ask for specifications" but you have certain expectations of what a car should behave like and what they gave you doesn't really qualify as a car no matter if their disclaimer was technically correct.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#392

Post by adql »

NohatCoder wrote: post #350
Scarblac wrote: post #336

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al. A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.

And that you can also lease servers directly from vendors like OVH so you don't even need to bother with the "drive to datacenter and install it" part. It's more expensive but still far cheaper than cloud

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#393

Post by TexanFeller »

m_mueller wrote: post #208
Paul-Craft wrote: post #189

Earlier quoted context omitted.

> Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon I'm still trying to figure out which one Aurora and Cognito fall under.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Aurora significantly modifies the internals of the DBs, particularly the storage layers. It also makes large changes to how memory is used for Postgres. Query plans can be quite different than with the vanilla version. Once you tune and create indexes based on Aurora's characteristics it's going to be a pain to retune for the unmodified version. Aurora also introduces nasty bugs that don't exist on the RDS version such as a memory leak I found was periodically restarting our master. The Postgres team produces highly reliable code, but I don't trust Aurora's hacks on top of it.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#394

Post by darkwater »

dontlaugh wrote: post #338
boxed wrote: post #112

Earlier quoted context omitted.

I built a little microservice on the side of my monolith for PDF creation. It used headless chrome and ghostscript to render html to a nice PDF. The problem I had with having that code inside the monolith was that it increased my docker image creation for deploys by a lot . And that code pretty much never changed anyway. I did feel a bit embarrassed having to make a microservice after having argued against them so mu…

You can produce multiple docker images from the same codebase quite easily. You can deploy and scale them separately. None of that requires separate repositories or expensive RPC instead of local function calls.

What's the difference if they are in one or two repo if they produce two artifacts that are separated? You *will* have network calls between the two, unless you are marrying yourself with a deployment/operational platform that can run the two artifacts together. (ok, there could be a few but I really don't see how this is just using a "monorepo" instead of a "multirepo")

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#395

Post by Scarblac »

jabradoodle wrote: post #355
Scarblac wrote: post #336

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

You also have to account for the fact your paying upfront the cost for the lifetime of the infra, vs paying monthly.

It's also possible to rent hardware in a rack and pay monthly, that's much cheaper than cloud services.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#396

Post by Scarblac »

gtirloni wrote: post #372
NohatCoder wrote: post #358

Earlier quoted context omitted.

They kind of need to be there anyway, physically maintaining servers turns out to be a miniscule part of the whole maintenance. If you really care about uptime you still need people on-call who can intervene as necessary.

It's not a minuscule part of a small company. I made the point that on-premises makes sense after a certain scale. Once you have on-premises you need people that know switches, routers, rackmount server, hardware, virtualization, etc, plus keeping all of that properly maintained (security patches, IaC, periodic updates, analyzing performance, making sure it's properly architected, etc). I often see people saying it's…

I don't know, I worked at a few companies that did this early in my career (early 2000s), and it was just the devs or the sysadmin of the office IT that did this sort of thing. There are lots of people who know enough about switches and routers to get them up and running.

Virtualization, IaC, analyzing performance, right architecture etc is all for later, when you've grown enough to need that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#397

Post by hobs »

NohatCoder wrote: post #350
Scarblac wrote: post #336

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al. A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.

I just spent the better part of two years advocating, pushing, and fighting for months to add new bandwidth to our datacenter.

Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other.

While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a midsize company ime.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#398

Post by harrisi »

andai wrote: post #353
boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

I haven't dealt with high traffic systems but isn't a few requests per second well within the capabilities of a $5 VPS?

There was a blog post and discussion here about this a few months ago: https://news.ycombinator.com/item?id=34676186

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#399

Post by jmillikin »

withinboredom wrote: post #313
jmillikin wrote: post #286

Earlier quoted context omitted.

1. Add a test with a time-bomb (such as a test certificate with 365-day duration), wait a year, and now your test fails without having changed. 2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing. 3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing. 4…

Hmm. We never had those specific issues. For (1), we had time bombs for sure, but those usually highlighted coverage issues. Dev ops would disable your test and tell you to fix your code. IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests. For (3), all de…

Yeah, all of those mitigations don't work as well when you've got thousands of engineers whose work would be blocked if some intern's badly written test blows up.

A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#400

Post by bewo001 »

siva7 wrote: post #203
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

The problem is more like that many people don't understand the tradeoffs and when to use microservices. This becomes even more obvious when you ask them what their current architecture is and what problems they hope to solve for that needs a transition to another architecture.

The reason for doing microservices I've been given by a two person developer team that had created a 15+ microservice single-server k8s monster was: 'this is how it is done today' Yeah, IT is like the fashion industry.
Post reply on HN