Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

241–250 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#241

Post by bootsmann »

BbzzbB wrote: post #194
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

Still waiting for python 3.11 on lambdas so must not be that big of a focus.

(They finally delivered 3.10 last month at least)

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#242

Post by kerkeslager »

Paul-Craft wrote: post #214
m_mueller wrote: post #208

Earlier quoted context omitted.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

> Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

I got a bit of a chuckle out of this. There's no way to move from Aurora to RDS short of... 2 minutes of actual work and a lot of waiting around due to the limitations of the hardware?

I get that it's not as easy as a literal button click, but this isn't vendor lock in.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#243

Post by sverhagen »

Paul-Craft wrote: post #214
m_mueller wrote: post #208

Earlier quoted context omitted.

Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.

Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

If you talk about no vendor lock in, and you'd want to take your database then to a competitor, like Google Cloud, Azure, or on-premises, wouldn't you exactly expect to do a SQL dump and reloading everything? To me, the one-click move from RDS to Aurora you describe is a nice shortcut, but it doesn't invalidate that you can still do the former if you wanted to move to the competitor. Vendor lock in seems more that you've architected your application against a system that only exists on AWS, like SQS or S3 (although, I guess, competitors offer compatible APIs for some of those, I'm not entirely read up on the state of things there).

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#244

Post by blowski »

helsinkiandrew wrote: post #237
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

> I wouldn't be surprised if it's taken down at some point ... Why? they're still using "AWS stuff" - EC2 and ECS etc. Serverless is a fraction of the services AWS offers. AWS actively promote ways of reducing customers bills. This article could be considered a puff piece for the AWS Compute Savings Plan: https://aws.amazon.com/savingsplans/compute-pricing/

Exactly. You could easily frame it as "if AWS seems expensive, you're using it wrong". That an internal team could get it so wrong is testament to how difficult it is to get right, but of course, there's a consultant for helping with that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#245

Post by goodpoint »

motbus3 wrote: post #156
lastangryman wrote: post #68

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another…

I think it is fine. There are scenarios were you need distributed and there are scenarios that you don't. IMO, distributed software is more practical for working development than for technical reasons. We all know from basic stuff that performing software comes from single structures that does not require packing and unpacking data But scaling large applications is hard, and it was much more expensive back then. Now…

[flagged]

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#246

Post by Hamuko »

411111111111111 wrote: post #223
Hamuko wrote: post #176

Earlier quoted context omitted.

I'm not really interested in the operational overhead that brings for these small services. Cost-wise they might be just about neck-and-neck, but at least I don't need to worry about the server going down, or having outdated software. Lambda gives me scaling, load balancing and redundancy for that $2.

> these small services I think you might be unintentionally arguing with a strawman, as everyone else here is talking about using monoliths instead of that. Few people want to administer a bunch of micro services themselves, but running a single service on a box is pretty low effort, even if you duplicate it for fail over/redundancy

By "using monoliths", do you mean bolting all code you write into a single runtime, even if they are not the same service? Because that's not what was in this article. Instead, they took Step Functions and a bunch of Lambda functions, and created a brand new monolith from that.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#247

Post by bottlepalm »

klabb3 wrote: post #226
bottlepalm wrote: post #179

Earlier quoted context omitted.

Every service needs it’s own database. Then you need to handle all the DLQ errors between services as well.

Devils advocate, but is it possible microservices need a shared feature-ful message passing layer, and good tooling, to be working well? Eg schema, auth, flow control, ttl, persistence, partitioning etc in the message layer, a la Kafka? I mean it’s kind of implied that microservices can only work if they can talk to each other, and “talking” is a lot more nuanced than we tend to think.

Comm requires massive overhead versus simply 'calling a function'. Calling a function doesn't fail. Maybe the code in it does, but not the call to the function itself.

This is why microservice costs often far outweighs the benefits, but they rarely consider the cost in their crusade to 'break up the monolith'

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#248

Post by bobsmooth »

sgtnoodle wrote: post #89
selcuka wrote: post #25

Earlier quoted context omitted.

They should've serialised bitmaps to JSON and used SQS instead. /s

Over 15 years ago now, I was an intern at Toyota. We were working with an in-house python based framework for doing cool/terrible drive-by-wire things with test cars. I had a project to work around a bottleneck of the framework. It could only process about 70 CAN frames per second before running out of CPU. The vehicle's CAN bus had several thousand per second, though. At the time I was able to fix the problem by add…

>They converted every 8-byte payload buffer into a canonical binary representation, i.e. ascii strings of 1's and 0's.

Honestly sounds like something I'd do but I've never programmed anything more dangerous than a toaster let alone a car.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#249

Post by IshKebab »

bootsmann wrote: post #241
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)

If you're sensitive to a 25% performance improvement, why not switch languages and get a 1000% performance improvement?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#250

Post by ImPleadThe5th »

bawolff wrote: post #10

I'm pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

I really feel like microservices primarily solve people/team organizing problems more than it solves any computing problems.
Post reply on HN