Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sharding Pinterest: How we scaled our MySQL fleet

engineering.pinterest.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: Sharding Pinterest: How we scaled our MySQL fleet

#81

Post by joantune »

luckycharms810 wrote: post #17
halayli wrote: post #12

Earlier quoted context omitted.

Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).

Technical debt doesn't necessarily arise from using the wrong (a.k.a pretty similar but with a few features less than what you need) tool. In my experience, it's more likely that technical debt arises from trying to use thew newest tool (which looks to be a silver bullet) rather than trusting your own experiences. It's surprising to me that people are willing to move to a completely different technology rather than e…

Although, by using an existing tool, as long as it fits your needs, you'll avoid reinventing the proverbial wheel, ofcourse at the expense of learning something new.. which might not be a bad thing..

Re: Sharding Pinterest: How we scaled our MySQL fleet

#82

Post by mhd »

needusername wrote: post #76
detaro wrote: post #33

Earlier quoted context omitted.

It also easily fits in a Postgres instance on a single host, if you only look at "can store X amount of data". And even if Postgres is slower, for the money you save in license costs you can buy a few beefy nodes extra.

And your engineers work for free.

Free, no. Less money than Oracle DBAs? (Yes.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#83

Post by joshenders »

ck2 wrote: post #38

PINTEREST: publish your public outgoing IPs so we can whitelist you and block the rest of amazon ec2 It is the responsible thing to do, otherwise other website bots can spoof you.

Hi there, Pinterest traffic engineer here. I've added a task to take care of this. Thanks for the suggestion!

Re: Sharding Pinterest: How we scaled our MySQL fleet

#84

Post by rakoo »

thezilch wrote: post #24
z3t4 wrote: post #19

Nice article and good explanations. But I still wonder why some people store blobs in a database rather then simple files on a file-system!?

There are a lot of things your FS is not going to give you (eg. transactions), and if you're going to have to query it anyway to get that, you might as well have your blob there too. For "small" blobs, say 1MB blobs: http://research.microsoft.com/pubs/64525/tr-2006-45.pdf

Even better from the SQLite page (https://www.sqlite.org/intern-v-extern-blob.html), comparing storing content in SQLite versus storing content in files and filenames in SQLite. The bonus of mixing both is that you get all niceties from a SQL store (transactions, using SQL to query...).

The takeaway is that for content < 20K you should use SQLite, and for content < 100K you should also use it with possibly some tuning to do.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#85

Post by ck2 »

joshenders wrote: post #83
ck2 wrote: post #38

PINTEREST: publish your public outgoing IPs so we can whitelist you and block the rest of amazon ec2 It is the responsible thing to do, otherwise other website bots can spoof you.

Hi there, Pinterest traffic engineer here. I've added a task to take care of this. Thanks for the suggestion!

I'm embarrassed I didn't say "please". But thank you!

Re: Sharding Pinterest: How we scaled our MySQL fleet

#86

Post by joantune »

mark242 wrote: post #56

Am I missing the obvious? They're using text columns as blob stores for JSON data? How on earth do you query that in MySQL? How do you run a secondary index, on say the user_id? Is it just one gigantic instance of Elasticsearch or Lucene?

HBase? no idea, I would also like to know that one.

Re: Sharding Pinterest: How we scaled our MySQL fleet

#87

Post by tapirl »

ScottWhigham wrote: post #79
tapirl wrote: post #78

Too many configs and maintenance. NoSQL is really better than SQL to scale your databases. It is just sad there is no one open source NoSQL db as good as Google BitTable.

NoSQL is really better than SQL to scale your databases. Downvoted for what is essentially your opinion presented as a fact. If you want to make such a statement, you'll need to give more context - either in the form of proof, experience, or "other". Just something would suffice really - anything. You'll find a lot of people agreeing with you if you add such context. Without context, it's nothing more than "Vanilla i…

NO context, it is just absolutely.
Post reply on HN