Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

341–350 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#341

Post by gtirloni »

Scarblac wrote: post #336
kypro wrote: post #282

Earlier quoted context omitted.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

Most companies will pay way more for the engineers maintaining their on-premises infrastructure than they would for AWS. On-premises still makes sense when you reach a certain scale. When you reach a certain scale.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#342

Post by EVa5I7bHFq9mnYK »

iLoveOncall wrote: post #212
EVa5I7bHFq9mnYK wrote: post #144

I guess what AWS sells is not servers, but software to manage them automatically, to load balance, to replicate etc. Once, in a short time, GPT can write such (pretty standard) software for you, Amazon will, too, go down.

Sure, ChatGPT will automate in a short time what tens of thousands of top engineers have built over a decade.

Of course not. It will help millions of small companies to write scripts so they won't need AWS anymore.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#344

Post by bhouston »

amluto wrote: post #158
bhouston wrote: post #101

I wish this was a good condemnation of microservices in a general use case but it is very specific to the task at hand. Honestly, the original architecture was insane though. They needed to monitor encoding quality for video streams so they decided to save each encoded video frame as a separate image on S3 and pass it around to various machines for processing. That is a massive data explosion and very inefficient. It…

> Another architecture that would work is to stream the encoded video from the encoding machines to other machines to decode and inspect. That would work as well. And again avoid the inefficiencies with saving and passing around individual images. No, that’s still a bad architecture. Bandwidth within AWS may be “free” within the same AZ, but it’s very limited. Until you get to very very large instance types, you max…

Ah. I was thinking that the encoding machines were not bandwidth limited but rather cpu limited as they were doing expensive encoding algorithms. So I was thinking the streams were streaming out at less than real time. I figured this was better than the dual/multi encode method I think they are now relying upon when all the detection code doesn’t fit on the same machine as the encoder.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#345

Post by credit_guy »

lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

I don't read it like that at all. Both solutions use the Amazon cloud. Only in one solution you distribute a lot of processes, just because it's possible, and easy to code. When they figured out that rampant distribution was costly, they put more thinking in keeping a lot of computation in the same place (so, "monolith", but still in the cloud). No surprise, they found great savings. If they hadn't, they wold not have written about it. But they had to put some (most likely major) effort into redesigning the application.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#347

Post by moonchrome »

datadeft wrote: post #293
moonchrome wrote: post #271

Earlier quoted context omitted.

Serverless has such bullshit insidious pricing that makes it seem like you're saving money only to figure out you're in shit once you're knee deep in it. For example you'll have to read fine print to find out that 256MB lambda will have the compute power of a 90s desktop PC because compute scales with memory. And to get access to "one core" of compute you have to use like 2GB of memory. Now you may say "serverless is…

All of things you are complaining about are well known facts that are clearly stated in the documentation. I don't care about what is the equivalent computing power in 90s desktop measurement because you cannot replace a lambda function with a 90s desktop, so it is pointless. The right approach is: I have a problem A that I can implement using AWS Lambda, AWS EC2 or your favourite DHH approved stack, how much of thes…

Can you point me to where this is clearly stated in the documentation ? I only found one reference as a passing note when I went searching for it. This would be a value displayed in the pricing calculator with a link to explanation if they were being honest.

90s CPU comparison is just to demonstrate how out of place it is with what people are used to even on lowest tier hosts with shared CPU cores. Low ram compute seems to be artificially limited to make low ram lambdas useful in very narrow use cases.

For reference I have a devops team in-company that deployed and maintained several AWS projects, including some serverless, even they were surprised at the low compute available at low RAM lambdas.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#348

Post by CubsFan1060 »

Paul-Craft wrote: post #214
m_mueller wrote: post #208

Earlier quoted context omitted.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

This can be done with DMS. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introd...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#349

Post by time4tea »

Microservices are just an architectural pattern, and like all patterns there are places where they are highly appropriate, and others where they are inappropriate.

Same for cloud, same for

If everything is a hammer you'll hurt your thumb/hand/arm.

At least now (for some time) the pattern is named, so broadly when talking about this sort of thing, the name conjures up the same/similar image in everyones heads.

There are all sorts of inputs to the choice of architectural patterns, including budget, scalability (up and down), criticality, security, secrecy, team skills and knowledge, preference, organisational layout, organisation size, vendor landscape, existing contracts, legal jurisdiction ....

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#350

Post by NohatCoder »

Scarblac wrote: post #336
kypro wrote: post #282

Earlier quoted context omitted.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al.

A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.

Post reply on HN