Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prime Video service dumps microservices, cuts AWS bill 90%

thestack.technology

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#11

Post by zmgsabst »

slekker wrote: post #8
zmgsabst wrote: post #6

This sounds like a bad first design: - didn’t just grab the whole video (or ~1min segment), but individual frames - put each detector in its own context, needing its own copy - created a StepFunction for each tiny piece This never would’ve passed design review when I worked at Amazon because the combinatorial blowout is obvious and predictable. I’m sure they switched to what was considered good practice even then: -…

If this was a system design interview, would this design be approved?

I’m guessing about their requirements.

They decided to go the extra step and decode locally to save the S3 bucket entirely — so maybe not, as my change still has that (wasteful) step.

But the questions “can we group these operations together to save on network?” and “will we hit API limits due to fanout?” are ones I’ve had (in other contexts) during design reviews. I think it’s weird they missed that, initially.

Fanning a video out per frame and per operation seems inherently problematic; batching seems the obvious answer. Design review is meant to address that (variety of) concern.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#13

Post by throwaway2990 »

zmgsabst wrote: post #10
throwaway2990 wrote: post #9

Earlier quoted context omitted.

This assumes the current functionality is the same as it was when it was originally created.

The article from Prime Video says that it has largely the same functionality, while outlining problems from the initial version.

But it also says it was never intended or designed to run at high scale.

They built something that solved the problem at the time. It worked. They outgrew it and evolved the architecture.

Now everyone’s on HN and Twitter claiming to be experts saying they would never do that and they know better.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#14

Post by AbbeSomething »

CPU’s are mind boggling fast compared to data transfer. There are exceptions but for the vast majority of workloads reducing data transfers is the easiest optimization.

But scale is the obvious key component, having CPU’s allocated to doing mostly nothing is such a waste.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#15

Post by selcuka »

throwaway2990 wrote: post #13
zmgsabst wrote: post #10

Earlier quoted context omitted.

The article from Prime Video says that it has largely the same functionality, while outlining problems from the initial version.

But it also says it was never intended or designed to run at high scale. They built something that solved the problem at the time. It worked. They outgrew it and evolved the architecture. Now everyone’s on HN and Twitter claiming to be experts saying they would never do that and they know better.

> But it also says it was never intended or designed to run at high scale.

Doesn't this statement itself indicate bad design? I mean, it's Amazon Prime Video. If you design a system for Prime Video you should design it to run at high scale.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#16

Post by ChicagoDave »

As I recall, https://icloudguru.com/ used serverless to serve their video lessons and said they were paying pennies.

I wonder what they're doing differently. I tried searching for the original article, but it's too ambiguous with their general offerings, so no luck.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#17

Post by throwaway2990 »

selcuka wrote: post #15
throwaway2990 wrote: post #13

Earlier quoted context omitted.

But it also says it was never intended or designed to run at high scale. They built something that solved the problem at the time. It worked. They outgrew it and evolved the architecture. Now everyone’s on HN and Twitter claiming to be experts saying they would never do that and they know better.

> But it also says it was never intended or designed to run at high scale. Doesn't this statement itself indicate bad design? I mean, it's Amazon Prime Video. If you design a system for Prime Video you should design it to run at high scale.

No. This is one component of Prime which is no longer a microservice. Indicating they still heavily use microservices.

They needed to solve a problem. It was probably a low priority thing they wanted, didn’t need it to scale as or be highly used. They designed and built it quickly.

It outgrew the architecture.

But during that time it solved the problem and made them money.

The requirement of software is to solve a problem. A second requirement is often to make money. It ticked both boxes.

That does not mean it’s bad design. This is something juniors and intermediate programmers forget. That something does not need to be designed perfectly on day one. It needs to work.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#18

Post by bob1029 »

acd wrote: post #7

Monoliths have local CPU L1-Cache speeds, distributed microservices have JSON desearlization,searlization, network round trips. Distributed stateless monoliths can be faster than microservice. Carbon foot print, cloud bill of too many microservices is probably high.

> Monoliths have local CPU L1-Cache speeds

This becomes particularly differentiating when you start looking at batching primitives. Getting that pipeline filled up can add another 10x+ to performance.

Stacking all of the multipliers in your favor, things do begin to look pretty silly for distributed crap: Moving from a network trip in the same datacenter to an L1 reference is about a million times faster. That is pretty direct terms - you could hypothetically claim your app is ~6 orders of magnitude faster than some competing solution if you do everything right and you aren't saturated on I/O.

All of this said, I am feverishly pushing most of our B2B product stack into "serverless" functions. We don't need 6+ orders of magnitude performance improvement for our business. We need compliance, standardization and stability above all else. We want to be able to point our fingers at someone else and say "you fix it".

Running everything in 1 process is something I have been very enthusiastic about so far, but it does have its downsides if raw performance (or simplicity) is not your primary objective. We are trying to sell our product up-market, and without the ability to say things like "our production database is N+3/multi-zone, uses TDE and the keys are stored in an HSM on Mars", we won't get much attention from those customers.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#19

Post by mixdup »

ChicagoDave wrote: post #16

As I recall, https://icloudguru.com/ used serverless to serve their video lessons and said they were paying pennies. I wonder what they're doing differently. I tried searching for the original article, but it's too ambiguous with their general offerings, so no luck.

>I wonder what they're doing differently

Largest thing that stands out to me is simply scale

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#20

Post by xiwenc »

dharmab wrote: post #2

Article's headline and text mention this was achieved by switching from Lambda to ECS, which makes a lot of sense. Lambda is expensive for frequently used services.

If i recall one of the biggest selling point of lambda (serverless) was the “infinite” scale. Sounds like the “cash cow” for cloud service providers when used in efficiently/inappropriately.

Besides the lower maintenance efforts.

Post reply on HN