Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

321–330 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#321

Post by adql »

cglong wrote: post #135
bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

My team owns an API monolith that hosts several completely unrelated endpoints. I keep thinking this would be a good candidate for breaking into microservices, but I do wonder if I'm buying into the hype.

That is probably the best use case for microservices (althought I'd just call them "services" at that part).

That assuming they are big enough to warrant that in the first place.

IMO if it is a dozen unrelated endpoints but all of that still takes less than dev/month to manage it's probably entirely wasted effort to "fix it"

And if any of them grows to warrant dev/month or more.... separate that one and only that one out of that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#322

Post by throwaway283bd »

kypro wrote: post #282
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

There's a middle ground between cloud and on-prem

You don't need to go all in on buying racks and other hardware, when you can rent servers at Hetzner at a cheaper cost than aws.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#323

Post by huksley »

AWS Step Functions are bad for so many reasons. Scaling, pricing, developer experience, etc.

It is clearly made by people who don't really understand (or does not care) how distributed workflows work.

And pricing are prohibiting to run it at scale. In my opinion it should be free to use, provided you glue together other AWS services with it.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#324

Post by jtc331 »

adql wrote: post #296
tough wrote: post #267

Earlier quoted context omitted.

What if you have a really big database? Still 2minutes?

Kinda depends if you can run replication from aurora to regular database

Aurora Postgres supports bog standard native Postgres logical replication.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#325

Post by lars512 »

boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

Tons of the pieces you mentioned are probably not that expensive to run for a small use case, given you're only charged on demand. The cost is really in the dev ops time and expertise to orchestrate the whole affair, and in the new ways it can break.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#326

Post by lastangryman »

boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

> AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it

I was back on AWS for the first time in a few years this week and the amount of new "upsell" prompts in the console is ridiculous. Spin up an RDS instance - "hey, would you like an Elasticache cluster too?". I think AWS are very aware of this behaviour and encourage it. Simplicity is not in their interest.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#327

Post by steveBK123 »

awaythrow98765 wrote: post #289
kypro wrote: post #282

Earlier quoted context omitted.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Cloud vendors also mostly sell minimum use packages for discounts in the range of 20 to 80% (called e.g. "committed use discount" or "compute savings plan"). Lots of businesses use those, because two-digit discou…

Yup, and you are paying the premium of cloud forever, which over some vanilla compute & storage can be a lot.

And cloud proponents pretend data center / rack space / server leasing doesn't exist either, for those trying to avoid large up front costs.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#328

Post by bob1029 »

I think serverless has its place, but this problem doesn't seem like a fantastic fit.

We are looking into serverless as a way to exhibit to our customers that we are strictly following certain pre-packaged compliance models. Cost & performance are a distant 2nd concern to security & compliance for us. And to be clear - we aren't necessarily talking about actual security - this is more about making a B2B client feel more secure by way of our standardized operating model.

The thinking goes something like - If we don't have direct access to any servers, hard drives or databases, there aren't any major audit points to discuss. Storage of PII is the hottest topic in our industry and we can sidestep entire aspects of The Auditor's main quest line by avoiding certain technology choices. If we decided to go with an on-prem setup and rack our own servers, we'd have to endure uncomfortable levels of compliance.

Put differently, if you want to achieve something like PCI-DSS or ITAR compliance without having to covert your [home] office into a SCIF, serverless can be a fantastic thing to consider.

If performance & cost are the primary considerations and you don't have auditors breathing down your neck, maybe stick with simpler tech.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#329

Post by hobs »

kerkeslager wrote: post #242
Paul-Craft wrote: post #214

Earlier quoted context omitted.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution. > Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employe…

If you can't stream changes and have to take downtime for a migration then you effectively have vendor lock in if you are serious enough about your database.

Physical Replication should be something any database can offer given its something cross database migrations used decades ago with no problem.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#330

Post by ripper1138 »

LASR wrote: post #274

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was "fixed". Some excerpts: > This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. My candid reaction: Seriously? WTF? I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and…

This is what L6 and L7 are building at Amazon, meanwhile in sys design interviews I’m being asked to design solutions for a gaming platform with 50M concurrent users.
Post reply on HN