Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

211–220 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#211
> The main scaling bottleneck in the architecture was the orchestration management that was implemented using AWS Step Functions. *Our service performed multiple state transitions for every second of the stream*(???), so we quickly reached account limits. Besides that, AWS Step Functions charges users per state transition.

This is so obvious in my head. I can't think of a single good reason where a SFN makes sense here.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#212

I guess what AWS sells is not servers, but software to manage them automatically, to load balance, to replicate etc. Once, in a short time, GPT can write such (pretty standard) software for you, Amazon will, too, go down.

Sure, ChatGPT will automate in a short time what tens of thousands of top engineers have built over a decade.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#213
post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I agree, my intuition would put it to 1% vs. 99% (difficult to quantify of course). I haven't yet seen a project/product which would need microservice architecture for technical reasons. If you need to scale, you can just scale monoliths (perhaps serving in different roles). The use case for microservice architecture is IMHO an organizational / high level architecture driven. I've worked in a big company (20K employe…

I'm developing an ML based sideproject. All the modern ML tools are written in python, which is a reality good language for it. However, it is an abysmal project for writing business logic and third party integrations, and if I have some free time I will split the whole thing into one Python and one Typescript service.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#214

Earlier quoted context omitted.

> Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon I'm still trying to figure out which one Aurora and Cognito fall under.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#216
post #192
post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I had to google what 'cargo culting' meant. But I laughed when I found out. https://en.wikipedia.org/wiki/Cargo_cult_programming#:~:text... .

Do you have a link to documentation on that method of highlighting text? Hard to search for due to the all the non-letter characters.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#217
post #215

Dead horse and all that but please just stick to Boring Tech, it is better for your mental health, not to mention your business, development velocity, defect rate, etc. Most importantly it's good for mental health though.

Not good for resume padding hype chasers. Especially the managerial types who never need to actual write the code.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#219
Over engineering at its best. I tend to see microservices as a doubled edged sword and in this case, there was no need for them.

Also, the pricing of AWS quickly goes up as you go from EC2 -> Fargate -> Lambda. I don't know why on earth someone would build microservices at the lambda-level.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#220
post #160

Earlier quoted context omitted.

Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in…

In your example, you compare Lambda against a separate monolith for handling the webhooks, but with a monolith wouldn't the comparison be between lambda and just adding a route and controller (or equivalent) to the monolith?

I'm more thinking of rewriting the application (a bunch of Lambda functions + API gateway + random bits and bobs) as a monolith and running that separately on an EC2 instance (or any other VPS).

In this article, they didn't bolt the serverless architecture onto another existing monolith, but rather rewrote the Step Functions and Lambda functions to be a single ECS task.

Post reply on HN