Live data from Hacker News

Netflix Billing Migration to AWS

techblog.netflix.com

91–100 of 126 posts

Re: Netflix Billing Migration to AWS

#91
post #22

Earlier quoted context omitted.

Is there a version of poe's law for hacker news / slashdot? A large company describes their very real efforts and shares their experience and knowledge, and the first comment is "Why not use instead?".

Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable. Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.

> Postgres is usually the first choice for applications like this

Do you have any data to substantiate this claim?

Re: Netflix Billing Migration to AWS

#92
Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence all relevant data was migrated to MYSQL.

Considering that Cassandra is not ACID compliant with her "eventual consistency", and that MySQL is notorious for corrupting data and not functioning correctly, I am compelled to wonder just what kind of people work at Netflix. And who gets the idea to go to AWS and pay the full virtualization on Linux performance penalty?

Now, I've done Oracle engineering at some very large databases (hundreds of millions of rows, OLTP and DWH), and I know that Oracle is a smoking fast database when the right people develop on it. Also makes me wonder what kind of code they had running, and what kind of people selected it, when they managed to gum up what is essentially the Bugatti Veyron of databases.

Given this information from "Netflix" I won't be considering them as a potential employer any time soon. It has to be a mess over there.

Re: Netflix Billing Migration to AWS

#93

Earlier quoted context omitted.

I work on the Netflix Billing Team. PostgreSQL was indeed a very attractive option, but we wanted to keep a path to Aurora open. When we were working on the migration, Aurora was still in beta, so instead of going to Aurora directly, we decided run our own MySQL instances on EC2.

Why would you go to RDS with such large amounts of data when AWS do not provide ability to get data out easily. If you moved away from AWS in the future for what ever reason your data is more or less stuck in AWS.

Amazon released their data migration service a while back, which allows you to transfer data in just about anyway you might want to, they'll migrate data between different RDS engines (MySQL to Postgres for example), and to databases outside AWS. They even support near real-time replication to database servers outside AWS, so you could hypothetically replicate your RDS instances to a fail over environment with another provider. There's very little risk of your data being locked in now.

[1] https://aws.amazon.com/dms/?nc2=h_mo

Re: Netflix Billing Migration to AWS

#95
post #92

Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence al…

The majority of the corruption issues with MySQL comes from using the innodb engine. AWS built their own MySQL engine called Aurora that I would be shocked if Netflix wasn't using. It's designed for distributed workloads and should be harder to corrupt.

I'll admit I'm confused about picking Cassandra as well, but not for the same reasons you are. They're only storing subscriber data (billing address, subscription type, etc). That data is going to remain static for months at a time. When changed the only potential problem that could occur is the billing process using old data, but I'm guessing their system is smart enough to try again in thirty minutes.

Oracle may be fast, but it's also expensive. This is billing, which means batched jobs running in the background- they don't care how long each individual transaction takes, and I'm positive it's going to be cheaper to roll up more servers to compensate than it is to pay Oracle's licensing fees.

Re: Netflix Billing Migration to AWS

#96
post #84
post #2

They used a DRBD replicated MySQL. Wonder why they used MySQL over PostgresSQL then.. Would be great to know if they looked that up aswell.

In the last 15 years every news (on /. and HN) that mentions MySQL is interrupted with such questions from Postgres fanboys. How about: MySQL is the very best solution for such tasks and Postgres isn't the universal holy grail? Wouldn't it be better to try to convert people over from (Postgres direct competitors) MSSQL, OracleDB, DB2 to Postgres?

Frankly, no; I've used all of Postgres, MySQL, MSSQL, and SQLite, and MySQL is the worst of the lot. No, PostgreSQL isn't a holy grain, but any user of MySQL would overall be better served by PostgreSQL; especially on AWS where the replication is so simple to set up, which IMO is the only good argument I've heard in favor of MySQL (but I hear even that is fraught with peril).

MySQL is so completely riddled with bugs and utterly baffling behavior — it is the PHP of relational DBs. What I remember off the top of my head includes attempting to subtract datetimes causing the punctuation to be removed from the datetimes and the resulting "integers" to be subtracted, FK integrity being violated in a number of easy-to-hit corner cases, the "utf8" encoding not being able to encode UTF-8 (and the default being latin1…), GROUP BY allowing obviously (i.e., catchable to the parser) broken queries, `SELECT * FROM table` on Thus far the only other tool I've seen come close to this level of insanity is MongoDB. The thing about a tool so willfully discards any sort of reasoned approach to its topic area is that the people who use it — who inevitably are not well versed in how a relational database works "in theory" — cannot derive from its behavior its rules, because its behavior is irrational, bordering on psychotic, and the people using it tweak query after query while having no understanding of why one query might work better until some abomination that someone usually works is crafted; `-- Don't touch`. A good tool — I believe, somewhere deep inside me — will teach the novice user. MySQL will not; it will drive you insane.

[1]: http://grimoire.ca/mysql/choose-something-else

Re: Netflix Billing Migration to AWS

#97
Thanks for writing this up, I'm just at the tail end of having re-architected the CloudFlare billing system, also a subscription system written in Java with a MySQL back-end, but fronted by a Go API that insulates the rest of the business from the internals of the billing system.

The blog post covers a lot of the high level stuff really well, but I'm interested to learn whether you experienced any issues along the way, what they were, how you dealt with them.

In CloudFlare's case our migration was made more complex by also adding PayPal, and changing our processing gateway. Both of which created risk that we've had to work hard to understand and mitigate, i.e. how different gateways may, with the same card, return different results, etc.

Re: Netflix Billing Migration to AWS

#99
post #95
post #92

Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence al…

The majority of the corruption issues with MySQL comes from using the innodb engine. AWS built their own MySQL engine called Aurora that I would be shocked if Netflix wasn't using. It's designed for distributed workloads and should be harder to corrupt. I'll admit I'm confused about picking Cassandra as well, but not for the same reasons you are. They're only storing subscriber data (billing address, subscription typ…

Amazon doesn't offer instances large enough on RDS, so they are managing their own.

Do you have any sources for the corruption comparison between Aurora vs InnoDB?

Re: Netflix Billing Migration to AWS

#100

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing…

How do you know Microsoft will not compete with you sometime next quarter? Did you anticipate the LinkedIn purchase with which Microsoft enters the job search market?
Post reply on HN