Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix Billing Migration to AWS

techblog.netflix.com

Post reply on HN
Print view View original

61–70 of 126 posts

Re: Netflix Billing Migration to AWS

#61

Post by tomphoolery »

partiallypro wrote: post #36

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing…

> Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins.

So you're saying Amazon is going to risk millions of dollars so they can make a few more bucks on video streaming, which is like 3 levels down from their primary business?

Prove it. I had an argument with my last company about this very issue. If Amazon's primary business was video delivery, then that would make a lot of sense. But where does Amazon's primary revenue stream actually come from? That's right, it's AWS. Amazon may be an excellent retailer, but they spend just as much money as they make on the shipping and fulfillment side to get shit on your doorstep faster and cheaper than anyone else out there. Each person that spends money on Amazon can only really spend a few hundred dollars per year. But even a small company that's entirely hosted on AWS, like 70% of the companies I've worked for, pays Amazon thousands of dollars per month for hosting. There's definitely more people than companies, but shipping stuff to people costs a lot more money than Amazon needs to pay in order to have your stuff hosted by them. Plus they do a lot of R&D into making their own systems faster and more efficient, eventually passing savings down when they re-work their pricing tiers.

Basically, my argument is/was that the whole idea of Amazon stealing your IP to make a few extra bucks on whatever they happen to be doing is totally bunk. Amazon is really in the infrastructure business, and if you're a video startup...you're NOT. And take it from someone who watched a company try and fail to build a competing private cloud with no budget and a skeleton crew...it's stressful and not fun at all.

The main reason to use multiple providers is, as you said, uptime and redundancy...and "not putting all your eggs in one basket", so to speak. It's an engineering, not political, decision. My last company could have probably saved themselves by moving everything to AWS and shutting down their Level3 internet-backbone connectivity and direct fiber from the office to the datacenter (which is the same technology AWS is using anyway, except they have an actual cloud API and not just a pile of servers), but they were too busy conflating this engineering/performance decision with one that must be made for political reasons.

Re: Netflix Billing Migration to AWS

#62

Post by tomphoolery »

GrinningFool wrote: post #58
hiuhsdf wrote: post #48

Earlier quoted context omitted.

Yeah, this decision makes no sense, AWS is not even cost-effective past a certain, relatively small, amount of resource usage. Netflix could have easily built their own infrastructure for this, plenty of smaller companies do it and save tons of money. The only thing that would make sense here is if Amazon offered them their services at a huge discount, which could be a move looking to either acquire Netflix in the ne…

Netflix started with their own infrastructure[1] and migrated it all to AWS. Presumably this was more cost-effective for them. [1] https://media.netflix.com/en/company-blog/completing-the-net...

it is WAY more cost-effective when you get to that kind of scale to use someone else's battle-tested APIs. the best part is that when that stuff breaks down, it's not your job to fix it.

Re: Netflix Billing Migration to AWS

#63

Post by NetTechM »

tomphoolery wrote: post #62
GrinningFool wrote: post #58

Earlier quoted context omitted.

Netflix started with their own infrastructure[1] and migrated it all to AWS. Presumably this was more cost-effective for them. [1] https://media.netflix.com/en/company-blog/completing-the-net...

it is WAY more cost-effective when you get to that kind of scale to use someone else's battle-tested APIs. the best part is that when that stuff breaks down, it's not your job to fix it.

Plus, no pricey salaries to have event response teams on the payroll.

Re: Netflix Billing Migration to AWS

#64

Post by ams6110 »

cpollard wrote: post #29
hetfeld wrote: post #12

Dropped Oracle, using MySQL. Why not use PostgreSQL instead?

I know regarding HIPAA, Aurora isn't supported but mySQL is. Maybe there's a reason why it's not HIPAA compliant that made it unusable for Netflix? And cost?

Where is the intersection between HIPPA and a back-office customer billing system?

Re: Netflix Billing Migration to AWS

#65

Post by chipperyman573 »

tomphoolery wrote: post #61
partiallypro wrote: post #36

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing…

> Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins. So you're saying Amazon is going to risk millions of dollars so they can make a few more bucks on video streaming, which is like 3 levels down from their primary business? Prove it. I had an…

>Amazon may be an excellent retailer, but they spend just as much money as they make on the shipping and fulfillment side to get shit on your doorstep faster and cheaper than anyone else out there. Each person that spends money on Amazon can only really spend a few hundred dollars per year. But even a small company that's entirely hosted on AWS, like 70% of the companies I've worked for, pays Amazon thousands of dollars per month for hosting

But there are a lot more people than businesses, and an order of magnitude more people than businesses who both need custom hosting infrastructure and choose AWS.

Re: Netflix Billing Migration to AWS

#66

Post by kbenson »

ww520 wrote: post #56
kbenson wrote: post #31

Earlier quoted context omitted.

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again. MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same…

> 1. data loss due to async nature of replication

That depends on what you mean by async. The replication itself is synchronous (statements cannot happen out of order), it's just not lockstep with disk writes and commits. I think it's more illustrative to say it's delayed.

> 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP

I'm not referring to multiple live masters, I'm referring to a set of servers where each is both master and slave to the other, and one live master server which gets the HA IP address. In that respect, there is no difference to a DRBD replicated setup. Clients just use the HA IP.

> 4. complication in adding or removing master.

I never found it that complicated, and I deployed it at least 5-6 times for multiple companies, and even with slightly different topologies (master/master where each master had an additional dedicated slave reserved for intensive read-only queries). What did you find complicated about it?

Re: Netflix Billing Migration to AWS

#68

Post by justinmk »

partiallypro wrote: post #36

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing…

> I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service.

This was directly addressed in a 2011 talk[1] by a director at Netflix:

> Competition drives cost down

> [...]

> We would prefer to be an insignificant customer in a giant cloud.

[1] https://justinmk.github.io/2016/03/17/selfish-cooperation.ht...

Re: Netflix Billing Migration to AWS

#69

Post by e12e »

tomphoolery wrote: post #61
partiallypro wrote: post #36

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing…

> Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins. So you're saying Amazon is going to risk millions of dollars so they can make a few more bucks on video streaming, which is like 3 levels down from their primary business? Prove it. I had an…

Good points. I'd just like to point out that, there are "political redundancy" reasons to diversify to multiple (cloud) providers: you could easily get in a billing dispute with a single vendor - if that means your entire business grinds to a halt, then that puts you in a very weak and vulnerable position.

I don't know if the extra effort of using eg. Google and AWS is worth in order to "stay up when AWS goes down" - but it might be worth it to stay up if/when AWS cuts you off for some reasons (quite possibly due to human error, billing, a take-down notice or other legal dispute etc).

None of that helps if you might accidentally find yourself on the wrong side of the "war on terror" by publishing news - in such a case all your US funds and assets might be frozen, and you would need a non-US presence in order to stay up while sorting out the potential error. But I suppose it's no worse than being subject to other kinds of arbitrary censorship...

Re: Netflix Billing Migration to AWS

#70

Post by wmfiv »

singlow wrote: post #55
cpollard wrote: post #29

Earlier quoted context omitted.

I know regarding HIPAA, Aurora isn't supported but mySQL is. Maybe there's a reason why it's not HIPAA compliant that made it unusable for Netflix? And cost?

What aspects of HIPAA does Aurora cause problems for which wouldn't be a problem for other storage engines? The Aurora engine RDS product supports all of the normal MySQL access controls, SSL in transit and at-rest encryption for the data, snapshots and backups.

I don't believe it's a technical problem. It just hasn't completed the internal AWS process for HIPAA certification to be added to their BAA. Postgres is in the same boat.
Post reply on HN