Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prime Video service dumps microservices, cuts AWS bill 90%

thestack.technology

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#21

Post by selcuka »

throwaway2990 wrote: post #17
selcuka wrote: post #15

Earlier quoted context omitted.

> But it also says it was never intended or designed to run at high scale. Doesn't this statement itself indicate bad design? I mean, it's Amazon Prime Video. If you design a system for Prime Video you should design it to run at high scale.

No. This is one component of Prime which is no longer a microservice. Indicating they still heavily use microservices. They needed to solve a problem. It was probably a low priority thing they wanted, didn’t need it to scale as or be highly used. They designed and built it quickly. It outgrew the architecture. But during that time it solved the problem and made them money. The requirement of software is to solve a pr…

> A second requirement is often to make money. It ticked both boxes.

In this case it caused 10x higher AWS bills when compared to proper design, so I'm not sure if it ticked the second box.

If you know that your product will absolutely outgrow the architecture soon (again, we are talking about Prime Video) it makes more sense to design it properly from the beginning.

> They designed and built it quickly.

> That something does not need to be designed perfectly on day one. It needs to work.

But they tried to design it perfectly on day one, with the wrong assumptions. A monolith would be easier to design and quicker to implement. Again, I am speaking for this special case (because of the scale of the company/product). This is not premature optimisation, it is common sense.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#22

Post by zamnos »

ChicagoDave wrote: post #16

As I recall, https://icloudguru.com/ used serverless to serve their video lessons and said they were paying pennies. I wonder what they're doing differently. I tried searching for the original article, but it's too ambiguous with their general offerings, so no luck.

Just serve and not transcode? Because it sounds like the Amazon Prime microservice was doing some heavy processing of uploaded video - breaking it into frames and applying some ML algorithms to them, and also the same with the audio. Sounds like even if you can fit in under the 10 min lamda execution timeout (assuming internal teams are limited to the same), that AWS Lambda/Step Functions isn't a generic job queue handler and is not infinitely parallel like AWS would want you to believe.

It's a bitl ike using an SQL database for streamed append-only logging. It'll work in test, with test amounts of data, but doesn't actually scale.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#23

Post by that_guy_iain »

I ended up having a weird debate on Reddit over this. For some reason people seem to think that there is a single better choice and that if you ever go from a monolith to microservices that the monolith failed or whatever.

One person even made the claim that microservices are easier to maintain and debug even tho a single code base that you can use tools on and easily keep in sync seems miles easier to maintain than microservices where you need to version your interfaces so older services continue to work while you upgrade newer versions.

Then they were talking about with a monolith it takes longer to deploy. I assumed their actual deploy tasks took longer, which I can kind of see. But it turns out they were talking about how often they deploy.

It seems nuts to me that people think switching to microservices from a monolith will suddenly mean their poor practices that got their monolith into a bad state will suddenly disappear. From my experience, it generally just means they develop a distributed monolith and have all the same problems with some extra problems.

I'm of the opinion that if you can't develop a good monolith system you can't develop a good microservices system. A good microservices system while it has it's benefits is, in my opinion, harder to architect, develop, and maintain than a microservices based system.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#24

Post by emodendroket »

beoberha wrote: post #3

Here’s a link to the actual blog post from Amazon: https://www.primevideotech.com/video-streaming/scaling-up-th... It’s worth noting that the team that wrote the article is just a small part of Prime Video. The headline can definitely be interpreted to mean the entire Prime Video service.

I feel that the term "monolith" being in the title has led everyone to draw the wrong conclusions.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#25

Post by emodendroket »

zamnos wrote: post #22
ChicagoDave wrote: post #16

As I recall, https://icloudguru.com/ used serverless to serve their video lessons and said they were paying pennies. I wonder what they're doing differently. I tried searching for the original article, but it's too ambiguous with their general offerings, so no luck.

Just serve and not transcode? Because it sounds like the Amazon Prime microservice was doing some heavy processing of uploaded video - breaking it into frames and applying some ML algorithms to them, and also the same with the audio. Sounds like even if you can fit in under the 10 min lamda execution timeout (assuming internal teams are limited to the same), that AWS Lambda/Step Functions isn't a generic job queue ha…

Writing every frame to S3 and reading them back in other steps was probably a major factor.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#26

Post by emodendroket »

AbbeSomething wrote: post #14

CPU’s are mind boggling fast compared to data transfer. There are exceptions but for the vast majority of workloads reducing data transfers is the easiest optimization. But scale is the obvious key component, having CPU’s allocated to doing mostly nothing is such a waste.

Well, except that's not necessarily an "easy" optimization. If you go all-out with the highly managed stuff it's kind of like plugging together very simple Legos.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#27

Post by sofixa »

acd wrote: post #7

Monoliths have local CPU L1-Cache speeds, distributed microservices have JSON desearlization,searlization, network round trips. Distributed stateless monoliths can be faster than microservice. Carbon foot print, cloud bill of too many microservices is probably high.

> distributed microservices have JSON desearlization,searlization,

It doesn't have to be JSON, protobuf is 10x faster. Still slower than in-memory/CPU cache of course.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#28

Post by janosdebugs »

that_guy_iain wrote: post #23

I ended up having a weird debate on Reddit over this. For some reason people seem to think that there is a single better choice and that if you ever go from a monolith to microservices that the monolith failed or whatever. One person even made the claim that microservices are easier to maintain and debug even tho a single code base that you can use tools on and easily keep in sync seems miles easier to maintain than…

Funnily enough, both things can be true depending on the context.

Microservices are easier to debug: sure, if you compare a single monolithic binary with no logs and trace points to a microservice system that runs on top of a fancy trace the heck out of everything service mesh. Logging is still hard.

Deployment time: the same reason why people wouldn't compile dependencies statically. It's easier to replace a .dll or .so than the entire application, less bandwith. However, mess up the versions and you end up having a really bad day. Same with microservices: sure, a tiny service is easy to deploy, but heaven forbid you deploy incompatible versions.

etc etc.

Microservices give people who don't have an architectural view/experience in building complex systems from scratch a framework to put their stuff in without thinking much about it. Good luck getting the same level of organization in a monolithic NodeJS application. It's hard, even seasoned veterans often get the abstraction layers wrong.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#29

Post by sofixa »

xiwenc wrote: post #20
dharmab wrote: post #2

Article's headline and text mention this was achieved by switching from Lambda to ECS, which makes a lot of sense. Lambda is expensive for frequently used services.

If i recall one of the biggest selling point of lambda (serverless) was the “infinite” scale. Sounds like the “cash cow” for cloud service providers when used in efficiently/inappropriately. Besides the lower maintenance efforts.

> If i recall one of the biggest selling point of lambda (serverless) was the “infinite” scale

From zero to "infinite", so very useful for highly variable or completely unknown workloads. Especially with the container version migrating away isn't a massive undertaking, so it's still very useful as a starting point.

Re: Prime Video service dumps microservices, cuts AWS bill 90%

#30

Post by marcus_holmes »

janosdebugs wrote: post #28
that_guy_iain wrote: post #23

I ended up having a weird debate on Reddit over this. For some reason people seem to think that there is a single better choice and that if you ever go from a monolith to microservices that the monolith failed or whatever. One person even made the claim that microservices are easier to maintain and debug even tho a single code base that you can use tools on and easily keep in sync seems miles easier to maintain than…

Funnily enough, both things can be true depending on the context. Microservices are easier to debug: sure, if you compare a single monolithic binary with no logs and trace points to a microservice system that runs on top of a fancy trace the heck out of everything service mesh. Logging is still hard. Deployment time: the same reason why people wouldn't compile dependencies statically. It's easier to replace a .dll or…

I think part of the problem is that people who are wildly enthusiastic about microservices haven't experienced DLL Hell and don't understand the problems that can happen.
Post reply on HN