Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

engineering.tumblr.com

Post reply on HN
Print view View original

1–10 of 28 posts

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#3

Post by ErrantX »

This is excellent stuff; it fits a niggling problem I have (i.e. I'm a better programmer than sysadmin, so managing massive data shards is a pain)

It's an elegant implementation. From a ~30 minute read through I reckon I can use it to replace our current "hacked up" solution in just a few hours.

Kudos Tumblr.

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#5

Post by salimmadjd »

This is awesome! Also, I was at MySql event at Oracle two days ago and I overheard MySql guys talking with the pinterest folks about their sharding and how MySql team was going to announce something soon and wanted to get the pinterest's team feedback. MySql is doing their scripting on Python. So for a python shop, their release might be more interesting.

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#8

Post by evanelias »

salimmadjd wrote: post #5

This is awesome! Also, I was at MySql event at Oracle two days ago and I overheard MySql guys talking with the pinterest folks about their sharding and how MySql team was going to announce something soon and wanted to get the pinterest's team feedback. MySql is doing their scripting on Python. So for a python shop, their release might be more interesting.

Cool, looking forward to seeing that! I love Python too, and readily admit it's perhaps a more frequent choice for this type of automation.

That said, I really grew to love Ruby over the course of this project, which is actually my first in the language. Ruby's open classes allowed me to write a pretty flexible plugin/callback system with very little code. Jetpants allows you to hook arbitrary methods in before or after any method in any Jetpants class, and these callbacks "stack" (with support for different priorities) so multiple plugins can hook-in to the same place.

Because every large site seems to tackle sharding slightly differently, I figured a nice plugin system was pretty important in order for anyone else to be able to use this :)

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#9

Post by mokkai »

danmaz74 wrote: post #2

It's so good to see companies open sourcing some of their technologies. They probably do so mostly to improve their image, but who cares, and I have to say... it works with me! Thanks tumblr.

No. They mostly do it to ensure that the software is actively maintained. When a home grown software reaches a certain level of maturity it makes sense to set a roadmap and release it in the public domain. More users + developers = profit for both the company and the public.

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#10

Post by danmaz74 »

mokkai wrote: post #9
danmaz74 wrote: post #2

It's so good to see companies open sourcing some of their technologies. They probably do so mostly to improve their image, but who cares, and I have to say... it works with me! Thanks tumblr.

No. They mostly do it to ensure that the software is actively maintained. When a home grown software reaches a certain level of maturity it makes sense to set a roadmap and release it in the public domain. More users + developers = profit for both the company and the public.

Yes, if you get there that's a win-win. I wonder though how many projects really get a community support after they are open sourced by a company.
Post reply on HN