Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

451–460 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#451

Post by emodendroket »

Ensorceled wrote: post #449
emodendroket wrote: post #447

Earlier quoted context omitted.

Sure, but you don’t really expect to swap databases with a huge data store without any downtime, do you? I’m not aware of any technology that makes that easy.

Then why are you defending the ridiculous "2 minute" thing? If the CEO asked how long a migration will take would you respond "2 minutes of engineering time"?

In that situation I would guess the one-click tool doesn’t really handle everything you’d need either so I don’t get what the point of the comparison is.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#452

Post by kerkeslager »

tough wrote: post #267
kerkeslager wrote: post #242

Earlier quoted context omitted.

> > Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution. > Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employe…

What if you have a really big database? Still 2minutes?

Yes, still "2 minutes of actual work and a lot of waiting around due to the limitations of the hardware" which is what I said. Perhaps try reading the whole sentence next time?

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#453

Post by kerkeslager »

Ensorceled wrote: post #339
manfre wrote: post #292

Earlier quoted context omitted.

I'm intentionally ignoring any of the sarcasm in your comment. The time needed for a db dump is always dependent upon the amount of data. This is true regardless of the db software or where it's running.

I can't think of any project I've worked on where the main data base could be backed up and restored to a different database in "2 minutes" The sarcasm was warranted.

If only you had read the whole sentence, you might have saved yourself a bit of righteous anger.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#454

Post by __turbobrew__ »

withinboredom wrote: post #436
jmillikin wrote: post #399

Earlier quoted context omitted.

Yeah, all of those mitigations don't work as well when you've got thousands of engineers whose work would be blocked if some intern's badly written test blows up. A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.

It's a learning experience. If everyone learns from it, it probably (most likely) won't happen again. Everyone learns how to write better tests. And, like I said, if you absolutely need to merge something right this exact second and it can't wait until someone disables the failing test (or you can't do it yourself for some reason), you can always merge it even with failing tests.

Average tenure at most companies is 2/3 years. You cannot rely on people learning from mistakes because there are always new people, you need to make it so they cannot make mistakes or if they make a mistake it isn’t going to block the whole company from executing.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#455

Post by withinboredom »

__turbobrew__ wrote: post #454
withinboredom wrote: post #436

Earlier quoted context omitted.

It's a learning experience. If everyone learns from it, it probably (most likely) won't happen again. Everyone learns how to write better tests. And, like I said, if you absolutely need to merge something right this exact second and it can't wait until someone disables the failing test (or you can't do it yourself for some reason), you can always merge it even with failing tests.

Average tenure at most companies is 2/3 years. You cannot rely on people learning from mistakes because there are always new people, you need to make it so they cannot make mistakes or if they make a mistake it isn’t going to block the whole company from executing.

Organizations learn, it gets embedded in the culture, tooling, and automation. I shared some of the rules that were embedded in our organization via style guides and onboarding.

There were a thousand automated checks to prevent you from doing the same thing as someone else that caused downtime in the past. It was virtually impossible to commit code that deleted/truncated a table, for example.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#456

Post by kerkeslager »

Ensorceled wrote: post #449
emodendroket wrote: post #447

Earlier quoted context omitted.

Sure, but you don’t really expect to swap databases with a huge data store without any downtime, do you? I’m not aware of any technology that makes that easy.

Then why are you defending the ridiculous "2 minute" thing? If the CEO asked how long a migration will take would you respond "2 minutes of engineering time"?

If you were the CEO and your engineer and someone said, "2 minutes of actual work and a lot of waiting around due to the limitations of the hardware", would you interpret that as "2 minutes of engineering time"?

Obviously I'm going to spend a lot more time communicating the details of the situation to the CEO of a company that is paying me, than I'm going to spend communicating in a Hacker News comment. But as it turns out, no amount of communication is going to be effective if people don't bother to read past the first opportunity they see to jump in with a correction, even if that means stopping reading mid-sentence.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#457

Post by Jochim »

steveBK123 wrote: post #327
awaythrow98765 wrote: post #289

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Cloud vendors also mostly sell minimum use packages for discounts in the range of 20 to 80% (called e.g. "committed use discount" or "compute savings plan"). Lots of businesses use those, because two-digit discou…

Yup, and you are paying the premium of cloud forever, which over some vanilla compute & storage can be a lot. And cloud proponents pretend data center / rack space / server leasing doesn't exist either, for those trying to avoid large up front costs.

I'm a cloud proponent because it means not having to sit through hours of meetings to deploy a $5/mo virtual machine.

It also means some poor fuck at AWS gets woken up in the middle of the night instead of me when things go to shit.

It absolutely comes at a cost, and might not be the right fit for an organisation that's absolutely on top of it's hardware requirements and can afford to divert resources from new development work. For the rest of us it saves a lot of dev hours that would have otherwise been spent in pointless meetings or debating the best implementation of whatever half-baked stack has oozed it's way out of the organisation in an attempt to replicate what's handed to you with a cloud solution.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#458

Post by Jochim »

NohatCoder wrote: post #350
Scarblac wrote: post #336

Earlier quoted context omitted.

> If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.

This. And I think people tend not to understand how little actual hardware they are paying for when using AWS et al. A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.

I worked for a global company that maintained it's own "cloud" of VMs that we'd use for development purposes.

They were entirely unusable.

Opening a relatively small file in notepad could take multiple minutes. OS click and typing response times were measured in seconds.

Despite wasting thousands of developer hours each year, they refused to upgrade their data center. Probably because doing so would have been a major budget fight that requires an executive to actually advocate for something instead of making their characteristic animalistic grunts of agreement.

For better or worse I haven't seen the same issue with cloud expenditure. It seems to be perceived as a necessary expense, rather than the engineering department getting ideas above their station.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#459

Post by Jochim »

dijit wrote: post #409
hobs wrote: post #397

Earlier quoted context omitted.

I just spent the better part of two years advocating, pushing, and fighting for months to add new bandwidth to our datacenter. Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other. While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a mi…

I counter your anecdote with mine! A cloud vendor (who will be nameless as I signed an NDA specifically that prevents me from disparaging them; but one of the big three) ran out of capacity for me and it was 3 months before they managed to fix it. -- that was with a couple million a month in spend. Cloud is still servers; you just depend on someone elses capacity management skills and you hope that there isn't a rush…

Barring exceptional circumstances, I don't have to fight that fight at the cloud provider though. Their business is more likely to be amenable to maintaining and expanding reasonable levels of capacity.

I have to deal with a grumpy finance guy that thinks my whole department is overpaid already, especially so if we might use the dreaded `CapEx` word.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#460

Post by Jochim »

easton wrote: post #299
datadeft wrote: post #280

Earlier quoted context omitted.

It is possible[1]. You are better off this way than the AWS "native" way. 1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...

The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.

I used to believe the same thing, I've began thinking it might now be false yet often repeated. AWS claims that:

> Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives.[0]

This[1] article shows almost no discernable difference in .NET cold start times between containerised and regular lambdas.

It's easy to imagine developers pushing up bloated images, slowing startup down and blaming docker/AWS for it.

[0] https://aws.amazon.com/blogs/compute/working-with-lambda-lay...

[1] https://www.kloia.com/blog/aws-lambda-container-image-.net-b...

Post reply on HN