Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix Billing Migration to AWS

techblog.netflix.com

Post reply on HN
Print view View original

31–40 of 126 posts

Re: Netflix Billing Migration to AWS

#31

Post by kbenson »

ww520 wrote: post #18

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though.

I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

Re: Netflix Billing Migration to AWS

#32

Post by thinkingkong »

Rodeoclash wrote: post #25
eterm wrote: post #22

Earlier quoted context omitted.

Is there a version of poe's law for hacker news / slashdot? A large company describes their very real efforts and shares their experience and knowledge, and the first comment is "Why not use instead?".

Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable. Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.

When you say usually you probably mean "currently, usually" or "in the last x years usually".

Netflix has super smart people. They arent just picking random tools off the shelf and implementing things without constraints for the sake of it.

Re: Netflix Billing Migration to AWS

#35

Post by Rapzid »

kbenson wrote: post #31
ww520 wrote: post #18

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).

Re: Netflix Billing Migration to AWS

#36

Post by partiallypro »

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins.

I feel like Apple's approach of utilizing multiple providers makes more sense (though they do this for uptime and redundancy.) Maybe I'm being a pessimist.

Re: Netflix Billing Migration to AWS

#37

Post by nemothekid »

mikikian wrote: post #16

Dropped Oracle, using MySQL. Why not use AWS Aurora instead?

FTA: While our subscription processing was using data in our Cassandra datastore, our payment processor needed ACID capabilities of an RDBMS to process charge transactions. We still had a multi-terabyte database that would not fit in AWS RDS with TB limitations.

Re: Netflix Billing Migration to AWS

#38

Post by kbenson »

Rapzid wrote: post #35
kbenson wrote: post #31

Earlier quoted context omitted.

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).

> This is assuming you are using all InnoDB tables(you likely should be).

Ah, that's probably it. ~2004/2005 InnoDB wasn't nearly as popular as it is today, IIRC (but it was there, and was used).

In any case, multiple master with failover setups always worked well in my experience, as long as you take care to track the replication state.

Re: Netflix Billing Migration to AWS

#39

Post by dexterdog »

yeukhon wrote: post #26
tzakrajs wrote: post #20

How many times is Netflix going to finalize its transition to AWS?

They won't 100% be on AWS, but probably close to 95%. Netflix owns its own media CDN.

So almost all of their bandwidth is handled outside of AWS. I would wager a guess that bandwidth is quite a bit more than 5% of their ops bill.

Re: Netflix Billing Migration to AWS

#40

Post by angersock »

thinkingkong wrote: post #32
Rodeoclash wrote: post #25

Earlier quoted context omitted.

Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable. Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.

When you say usually you probably mean "currently, usually" or "in the last x years usually". Netflix has super smart people. They arent just picking random tools off the shelf and implementing things without constraints for the sake of it.

Right, which is why asking how they came to that decision (instead of using a DB dating back over three decades of development) is useful and reasonable.
Post reply on HN