Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

281–290 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#281

Post by datadeft »

abrookewood wrote: post #73
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

Yep, expect the Lambda team to raise hell.

Why would that be? Somebody figured out that AWS Lambda is not the answer to every single question?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#282

Post by kypro »

BbzzbB wrote: post #194
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses.

If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost.

With cloud if demand drops you can scale up and down as needed. Helping customers cut costs during difficult times makes sense since those customers are more likely to survive and stay with you through good times.

So in context I think this article makes sense since long-term sustainable growth of AWS should be linked with the growth of their customers' businesses.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#283

Post by boredumb »

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month when the entire thing could run on a few nodes and even a managed RDS instance for a thousand bucks a month. I would argue at this point that early optimization of architecture is astronomically worse than even* your co-worker that keeps turning all of your non-critical, low-volume iterable functions into lanes to utilize SIMD instructions.

Some irony in my anecdotal experiences is that most places that don't have the traffic to justify the cost of these super distributed service architectures also see a performance penalty from introducing network calls and marshaling costs

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#284

Post by kristopolous »

jddj wrote: post #88
kristopolous wrote: post #55

Earlier quoted context omitted.

There's a bunch of things I'd like to have them do. If they could span across machines like clusters that would be amazing. If I could trivially package them up and deploy them locally with intrinsically less effort and wall-time then the old way, that'd be amazing. If I could somehow get the horizontal scaling promises and redundancy as some kind of built-in, like I can with say, memcache, that's be cool. If I could…

Did you ever play with the Erlang VM?

Absolutely. It's pretty brilliant. I thought about mentioning it. I don't hate it. It's a bit exotic to trust with the likes who tend to fill the ranks of development teams which try to force every language into looking like C++ of Java but I dunno, send them off to a retreat in the mountains to microdose and take lessons from an erlang yogi for 12 weeks and have them come back.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#285

Post by bobsmooth »

Garlef wrote: post #109
quickthrower2 wrote: post #49

Earlier quoted context omitted.

Waiting for the next wave of tech misuse due to LLMs and ML!

There's a lot of room for "chinese-room" automation: Write a verbal description of your cloud function and let the LLM simulate the execution. Very cheap to develop. Very expensive to execute.

Prompt: Is the following number even or odd...

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#286

Post by jmillikin »

withinboredom wrote: post #180
__turbobrew__ wrote: post #140

Earlier quoted context omitted.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

1. Add a test with a time-bomb (such as a test certificate with 365-day duration), wait a year, and now your test fails without having changed.

2. Add a test with a network dependency, and when that dependency is slow / down / turned off, the test starts failing.

3. Add a dependency on a third-party Github repo that clones from `main`, and the next time some dev touches a file in that repo your test starts failing.

4. Add a test that allocates memory in proportion to size of the codebase (e.g. because it tries to build a giant in-memory tarball of all the .mp4 assets). Eventually it will get flaky when it starts scraping up against the build machine's limit. Extra fun if your builds run without defined memory limits on machines of different sizes.

In a monolithic build, there's all sorts of ways for a single person to cause other teams' tests to fail, even months or years after they've left the company. Some of them can be prevented mechanically (such as by running tests without network access), but a lot come down to "tell them to stop doing that".

That's why big companies never run one build per repo.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#287

Post by datadeft »

bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

Microservices make sense for a lot more than 5%. If fact I think it is much closer to the 80/20, 80% working on serverless, 20% not. Video streaming obviously not going to work on AWS Lambda to begin with.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#288

Post by citrin_ru »

__turbobrew__ wrote: post #140
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

1. Monolithic app and VCS mono-repo are orthogonal and can be mixed in different combinations (with different tradeoffs).

2. How about an old way of blocking deployments less: parts of this monolithic apps developed as libraries with a stable API, a new version of a library released only after its own tests has passed, next you can increment dependency version in the monolith and run integration tests, if they failing you can revert to the old library version and still go ahead with the deployment (if you don't depend on something added in the latest library release). If you depend on this new feature and a component providing it is broken micro-services would not help you.

> If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.

And here lies a very important problem IMHO - many (if not majority) of organizations (which do at least some software development) have less than 100 software developers but the industry best practices (which include micro-service architecture) are defined by FAANG-sized organizations and at least some of these practices are sub-optimal for small shops.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#289

Post by awaythrow98765 »

kypro wrote: post #282
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost.

Cloud vendors also mostly sell minimum use packages for discounts in the range of 20 to 80% (called e.g. "committed use discount" or "compute savings plan"). Lots of businesses use those, because two-digit discounts are real money, but they might find themselves in the same spot as with physical hardware they don't need...

Post reply on HN