Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling up the Prime Video audio/video monitoring service and reducing costs

primevideotech.com

Post reply on HN
Print view View original

431–440 of 526 posts

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#431

Post by gtirloni »

Scarblac wrote: post #396
gtirloni wrote: post #372

Earlier quoted context omitted.

It's not a minuscule part of a small company. I made the point that on-premises makes sense after a certain scale. Once you have on-premises you need people that know switches, routers, rackmount server, hardware, virtualization, etc, plus keeping all of that properly maintained (security patches, IaC, periodic updates, analyzing performance, making sure it's properly architected, etc). I often see people saying it's…

I don't know, I worked at a few companies that did this early in my career (early 2000s), and it was just the devs or the sysadmin of the office IT that did this sort of thing. There are lots of people who know enough about switches and routers to get them up and running. Virtualization, IaC, analyzing performance, right architecture etc is all for later, when you've grown enough to need that.

> Virtualization, IaC, analyzing performance, right architecture etc is all for later, when you've grown enough to need that.

Yeah, I think it might be a different perspective about when that all should be done.

I tend to do that right from the beginning because I often see it snowball later on and nobody ever fixes it or does it "properly" (in my opinion, possibly not the right one).

But that's a good point, no doubt.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#432

Post by drw85 »

boredumb wrote: post #283

AWS has a great business model of people over "optimizing" their architecture using new toys from amazon and being charged through the nose for it. It's amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month whe…

I actually worked on an Azure based project recently and it was very similar.

It was a small semi static contact form that was deployed on 27 web apps (9 services x 3 environments) and used a NoSQL storage, redis, serverless stuff, etc.

Insanely complex deployment process, crazy complexity and all over the place.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#433

Post by drw85 »

__turbobrew__ wrote: post #140
YetAnotherNick wrote: post #114

Earlier quoted context omitted.

Microservices works better if you don't trust other team. While having trust seem like a basic thing, this is absolutely not the case for a lot of companies. With microservices, it is easy to see services which are down or have high error rate or latency, have clear API contract and call out the team for breaking API contract, and assign cost for which the teams have incentive to reduce, or at least not increase it.

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred…

Aren't you just hiding the flaws in the overall system by confiding the tests to your microservice?

That's my experience at least. Things still break, you just notice it later.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#434

Post by shri_krishna »

dragonwriter wrote: post #152
shri_krishna wrote: post #106

The only time it makes sense to use edge/serverless anything is lightweight APIs and rendering HTML to end users so they get the page loaded as quickly as possible. That's the only use case good for edge. And any supporting infra that can help deliver rendered pages asap (like kv store on the edge for storing sessions, lightweight database on the edge for user profile data, queues etc). Anything that requires decent…

> The only time it makes sense to use edge/serverless anything is lightweight APIs and rendering HTML to end users so they get the page loaded as quickly as possible. That’s the only use case good for edge. Serverless and edge aren’t the same thing.

Nope. Edge is just serverless that is closer to your user to reduce the number of network hops. Both are essentially the same when it comes to technical functionality. They run on limited resources and should not be used for intensive workloads.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#435

Post by dragonwriter »

shri_krishna wrote: post #434
dragonwriter wrote: post #152

Earlier quoted context omitted.

> The only time it makes sense to use edge/serverless anything is lightweight APIs and rendering HTML to end users so they get the page loaded as quickly as possible. That’s the only use case good for edge. Serverless and edge aren’t the same thing.

Nope. Edge is just serverless that is closer to your user to reduce the number of network hops. Both are essentially the same when it comes to technical functionality. They run on limited resources and should not be used for intensive workloads.

Edge compute is serverless, but most serverless is not edge.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#436

Post by withinboredom »

jmillikin wrote: post #399
withinboredom wrote: post #313

Earlier quoted context omitted.

Hmm. We never had those specific issues. For (1), we had time bombs for sure, but those usually highlighted coverage issues. Dev ops would disable your test and tell you to fix your code. IRT (2), network dependencies were forbidden in-general. Over any long enough timespan, the rate of failure is 100%. If you wanted to use the network, you had to consider the failure case and handle it in your tests. For (3), all de…

Yeah, all of those mitigations don't work as well when you've got thousands of engineers whose work would be blocked if some intern's badly written test blows up. A monolithic build means that your ability to develop and deploy your team's code is dependent on every other team. As the number of teams gets larger, that multiplier really hurts.

It's a learning experience. If everyone learns from it, it probably (most likely) won't happen again. Everyone learns how to write better tests. And, like I said, if you absolutely need to merge something right this exact second and it can't wait until someone disables the failing test (or you can't do it yourself for some reason), you can always merge it even with failing tests.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#437

Post by shri_krishna »

dragonwriter wrote: post #435
shri_krishna wrote: post #434

Earlier quoted context omitted.

Nope. Edge is just serverless that is closer to your user to reduce the number of network hops. Both are essentially the same when it comes to technical functionality. They run on limited resources and should not be used for intensive workloads.

Edge compute is serverless, but most serverless is not edge.

You are just getting unnecessarily pedantic here. I was talking about computing resource usage. Both are the same when it comes to resource consumption being limited.

It is like saying Oracle/Postgres/MySQL/MSQL. If you say they are different in some X functionality, yeah duh they are different in X functionality. However, they are all SQL databases.

Same way, Edge/Serverless is both running on limited compute resources (which is the point of the article and point I was making). Both differing in functionality X (of latency/closeness to your user) has nothing to do with either the article or my answer.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#438

Post by briffle »

kypro wrote: post #282
BbzzbB wrote: post #194

Earlier quoted context omitted.

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

I know you're joking around, but no, as they also explained a benefit of cloud (and therefore using AWS) is that it can scale flexibly with their customers' businesses. If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost. With cloud if demand drops you can scale up a…

We have some DB servers that occasionally need to do very large batches of transactions. They run all month with a couple of CPU's, and a small amount of ram to make sure they are 'caught up' with production, and before the batches are run, get shutdown, and changed to 32 or 64 CPU monsters. An hour or two later, they go back to the 2 cpu servers again. In a non-cloud shop, we would have to size our hardware for that maximum batch size.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#439

Post by sgtnoodle »

bobsmooth wrote: post #248
sgtnoodle wrote: post #89

Earlier quoted context omitted.

Over 15 years ago now, I was an intern at Toyota. We were working with an in-house python based framework for doing cool/terrible drive-by-wire things with test cars. I had a project to work around a bottleneck of the framework. It could only process about 70 CAN frames per second before running out of CPU. The vehicle's CAN bus had several thousand per second, though. At the time I was able to fix the problem by add…

>They converted every 8-byte payload buffer into a canonical binary representation, i.e. ascii strings of 1's and 0's. Honestly sounds like something I'd do but I've never programmed anything more dangerous than a toaster let alone a car.

Horribly inefficient code is a wonderful thing at a small scale. The faster you solve your problem, the sooner you can solve the next problem.

I once threw together a mylar balloon helium blimp in the shape of a Dragon space capsule. My goal was to fly it over the cafeteria crowd at SpaceX during the C2 launch. For control, I used the PCB of a travel wifi router. I soldered three small DC motors to its LED outputs. The embedded software consisted of something like:

nc -l -u -p 10000 | bash

I then connected my laptop to the access point and ran a python script that would send UDP packets containing shell commands to toggle the LED GPIO pins based on arrow keypresses.

The crowd really enjoyed the novelty. After the excitement was over, I flew it around some more in the cafeteria. Elon Musk walked up to it floating in the air, paused for a few seconds, then looked around the room trying to find the operator. I was just like any other employee hanging out at a table casually typing on my laptop, though.

Good times. On my last day there I still had a helium tank under my desk. So, I filled up a life-sized Elmo balloon (a left over prototype), then let it float up into the rafters of the office. It was presumably up there for a month or two.

Re: Scaling up the Prime Video audio/video monitoring service and reducing costs

#440

Post by adql »

runlaszlorun wrote: post #420
adql wrote: post #402

Earlier quoted context omitted.

> It's not really the same spot in that your paying monthly rather than upfront. Devs tend to think about total $, the business/accountants do care about Opex vs Capex. You can buy physical servers in leasing ,turning it into opex You can also rent them for little bit extra via managed dedicated servers from vendors like OVH.

I think this point isn’t made often enough. Not going with a big cloud provider def doesn’t mean that you need to buy physical servers and build an on-prem data center.

And other point I also seen used to lie about cloud cost is saying you save so much on engineers.

...while forgetting to have sane on-call rotation for cloud you also need at least 3 people on that rotation that are also clued in on cloud operation enough. Sure they can be "developers" but if your app architecture requires so little maintenance and flea removal that they are not doing ops jobs much, chances are so would it in either rented or dedicated server env.

Post reply on HN