Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sharding Pinterest: How we scaled our MySQL fleet

engineering.pinterest.com

Post reply on HN
Print view View original

21–30 of 87 posts

Re: Sharding Pinterest: How we scaled our MySQL fleet

#21

Post by schneems »

pbreit wrote: post #10

Is it surprising that Pinterest is on MySQL? Has Postgres finally usurped the position of default DB for startups?

Came here to say the same. It makes sense that older companies are locked into their database as migrating would be too difficult like Facebook. I though Pinterest was a relatively young company and I'm surprised it chose mysql. Instagram is build on top of postgres and was founded in 2010, pinterest was founded in 2009.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#24

Post by thezilch »

z3t4 wrote: post #19

Nice article and good explanations. But I still wonder why some people store blobs in a database rather then simple files on a file-system!?

There are a lot of things your FS is not going to give you (eg. transactions), and if you're going to have to query it anyway to get that, you might as well have your blob there too.

For "small" blobs, say 1MB blobs: http://research.microsoft.com/pubs/64525/tr-2006-45.pdf

Re: Sharding Pinterest: How we scaled our MySQL fleet

#25

Post by meritt »

halayli wrote: post #6

This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".

While that would have made scaling easier they would then run into a different set of massive problems because Cassandra isn't a relational database.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#26

Post by nemothekid »

halayli wrote: post #6

This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".

Even today I'm not sure I'd recommend Cassandra for his use case. A stated requirement he needed was:

>Support asking for N number of Pins in a board in a deterministic order (such as reverse creation time or user specified ordering). Same for Pinner to likes, Pinner to Pins, etc.

This can be a pain to model in Cassandra. It would require denormalizing for every key you wanted to order on, and it generally makes updating data a pain.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#27

Post by needusername »

sarnowski wrote: post #4
boomzilla wrote: post #2

It seems mysql (and hopefully postgresql sometime soon) with custom sharding logic in the app layer still hits the sweet spot for scaling to the order of 100M users. With some thoughts going into designing an appropriate data model and sharding logic, certain join queries can be delegated to the databases too.

Hi, at Zalando, we are scaling all of our core businesses with PostgreSQL. Depending on your dataset, it can be fairly easy to shard your data for a horizontal scale-out (think of independent customer datasets). We have lots of databases that we scale horizontally to much bigger numbers. But, we also developed several tools that makes working with shards mostly transparent. Did not find a better source but one way we…

5 TB easily fits on a single Oracle instance on a single host.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#29

Post by econner »

thewarrior wrote: post #22

I know nothing about scaling databases but this reminds of Amazon's Dynamo DB object store but without the consistent hashing trick .

Dynamo is very complex, beyond consistent hashing. It also uses (or used) gossip and lots of tricks (hacks) to gracefully handle adding and removing nodes and distributing data to new nodes, etc. This uses a simple partitioning scheme albeit it does have its own consistency issues.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#30

Post by jlongtine »

nemothekid wrote: post #26
halayli wrote: post #6

This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".

Even today I'm not sure I'd recommend Cassandra for his use case. A stated requirement he needed was: > Support asking for N number of Pins in a board in a deterministic order (such as reverse creation time or user specified ordering). Same for Pinner to likes, Pinner to Pins, etc. This can be a pain to model in Cassandra. It would require denormalizing for every key you wanted to order on, and it generally makes upd…

I don't think that the schema here allows you to do this kind of query even in MySQL. They are throwing everything in that blob, which makes it pretty difficult to sort on anything that's in the blob. So, I think they would have to do what you are describing, denormalize everything... or get all the data and do the sort in the app. Which is precisely what would be required with Cassandra or another NoSQL system. Am I missing anything that would invalidate what I've just said?
Post reply on HN