Live data from Hacker News

Sharding Pinterest: How we scaled our MySQL fleet

engineering.pinterest.com

41–50 of 87 posts

Re: Sharding Pinterest: How we scaled our MySQL fleet

#41
post #39
post #38

PINTEREST: publish your public outgoing IPs so we can whitelist you and block the rest of amazon ec2 It is the responsible thing to do, otherwise other website bots can spoof you.

What's the problem with other bots appearing as pinterest?

Simple economics: they get some value from having their content on Pinterest. There is no value (that they could see) from having their content scraped by other bots.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#42
post #15
post #12

Earlier quoted context omitted.

Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).

The mapping tables are used to store relationships between objects, not to define shard locations. The shard mapping is encoded in the object id. In any case, I would have a knee jerk reaction to not trust any database system that hit 1.0 a month before I wanted to start using it, especially for mission critical core data.

For this very reason many projects nowadays jump directly from 0.x to 2.x :) 2 is the new 1.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#43
post #39

Earlier quoted context omitted.

What's the problem with other bots appearing as pinterest?

Simple economics: they get some value from having their content on Pinterest. There is no value (that they could see) from having their content scraped by other bots.

If that's the only reason then I hope pinterest doesn't publish their IPs.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#44

Isn't it unsafe to expose internal database IDs to external clients? I would have generated a second GUID for public view.

But how do you ensure that it is unique?

You will have a mapping table: Public_Guid, Private_ID

Every request has to convert the public, non-guessable GUID into the private ID that will be used in the lookup query.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#46

Earlier quoted context omitted.

Came here to say the same. It makes sense that older companies are locked into their database as migrating would be too difficult like Facebook. I though Pinterest was a relatively young company and I'm surprised it chose mysql. Instagram is build on top of postgres and was founded in 2010, pinterest was founded in 2009.

I believe Facebook had enough talent capable to choose their DBs carefully. And I am pretty sure MySQL replication had a lot to do with it. As having people who already knew how to finetune MySQL for big loads. BTW, believe it or not, Wikipedia once run on Postgresql, but was later migrated to MySQL (and currently they use MariaDB).

Proof-link? or lie. (pg fanboy here)

Re: Sharding Pinterest: How we scaled our MySQL fleet

#49
post #38

PINTEREST: publish your public outgoing IPs so we can whitelist you and block the rest of amazon ec2 It is the responsible thing to do, otherwise other website bots can spoof you.

Agreed, or implement RDNS verification like the major search engines do.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#50
post #31

Earlier quoted context omitted.

Came here to say the same. It makes sense that older companies are locked into their database as migrating would be too difficult like Facebook. I though Pinterest was a relatively young company and I'm surprised it chose mysql. Instagram is build on top of postgres and was founded in 2010, pinterest was founded in 2009.

Other than Oracle being a part of the equation MySQL is still an open source database and too quote the article: "Aside: I still recommend startups avoid the fancy new stuff — try really hard to just use MySQL. Trust me. I have the scars to prove it." In many ways, Postgres is cutting edge with it's features and capabilities. I see lots of updates from Postgres that include new SQL features (json objects) - conversel…

Well, that's exactly the thing about Postgres.

It is not stuck in the stone age, yet extremely reliable. Having worked with a number of MySQL (main fork) and Postgres installations in the past 7 years I observed a number of crashes and corruptions with MySQL (not counting the glory MyISAM days) while having _none_ with Postgres.

I should confess that that experience made me very biased in favour of Postgres. My default thinking about Postgres now is: if it doesn't have something MySQL has, it is either for a reason or they are taking their time to do it the _right_ way.

Post reply on HN