Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

engineering.tumblr.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#13

Post by evanelias »

joshjhacker wrote: post #12

Is there a mailing list for jetpants so we can ask questions? For example how should my application connect to mysql? Should it connect directly to mysql or to a jetpants server and how?

Good call -- we'll definitely set one up if there's sufficient need. Until then, feel free to email me; my email address is in the gemspec. I'll write up an FAQ once I have enough questions answered.

re: your immediate question, you still connect to MySQL as normal. Jetpants isn't a server, middleware, framework, or ORM. Rather, it's a toolkit. Jetpants has a command suite that you can use to run its built-in functionality, but it's also a Ruby gem that you can integrate into custom scripts however you'd like.

The functionality is all geared towards managing large DBs (importing/exporting lots of data quickly, copying files quickly, etc) and managing large numbers of servers (promoting/demoting masters and slaves, adding new machines to a pool, rebalancing a shard).

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#15

Post by evanelias »

Andys wrote: post #14

My quick reading of this is that its suited for databases that don't change much (or at all) once the data is inserted, and not as much for apps that need to keep strong ACID compliance with guaranteed referential integrity.

We handle many thousands of write queries per second at Tumblr, and we use Jetpants to manage our entire MySQL topology, so trust me when I say the data changes quite often :) You can edit your existing posts on Tumblr, unlike on several other prominent social sites.

Definitely please let me know how you got that impression though -- I'm happy to improve confusing things in the docs.

As for ACID compliance: Jetpants is a toolkit for MySQL / InnoDB, and doesn't really impact the referential integrity guarantees of those systems any more or less than other partitioning schemes. MySQL is inherently not a distributed system, for better or worse.

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#16

Post by joshjhacker »

evanelias wrote: post #13
joshjhacker wrote: post #12

Is there a mailing list for jetpants so we can ask questions? For example how should my application connect to mysql? Should it connect directly to mysql or to a jetpants server and how?

Good call -- we'll definitely set one up if there's sufficient need. Until then, feel free to email me; my email address is in the gemspec. I'll write up an FAQ once I have enough questions answered. re: your immediate question, you still connect to MySQL as normal. Jetpants isn't a server, middleware, framework, or ORM. Rather, it's a toolkit. Jetpants has a command suite that you can use to run its built-in functio…

Thanks!

Another question - we are using django so are considering Postgres since there is a python connection pool available. Could Jetpants be potentially used for Postgres? ie how much of the functionality is Mysql specific?

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#17

Post by evanelias »

joshjhacker wrote: post #16
evanelias wrote: post #13

Earlier quoted context omitted.

Good call -- we'll definitely set one up if there's sufficient need. Until then, feel free to email me; my email address is in the gemspec. I'll write up an FAQ once I have enough questions answered. re: your immediate question, you still connect to MySQL as normal. Jetpants isn't a server, middleware, framework, or ORM. Rather, it's a toolkit. Jetpants has a command suite that you can use to run its built-in functio…

Thanks! Another question - we are using django so are considering Postgres since there is a python connection pool available. Could Jetpants be potentially used for Postgres? ie how much of the functionality is Mysql specific?

The core functionality is currently very MySQL-specific. In theory a plugin could override a bunch of methods to target Postgres, and maybe even Redis or other persistent data stores with replication and import/export functionality. It would be a lot of work though.

I also made the mistake of putting "mysql" in the names of a few methods. At some point soon I'll change those to more generic names, and alias the old names to the new generic ones.

Re: Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

#19

Post by plasma »

evanelias wrote: post #17
joshjhacker wrote: post #16

Earlier quoted context omitted.

Thanks! Another question - we are using django so are considering Postgres since there is a python connection pool available. Could Jetpants be potentially used for Postgres? ie how much of the functionality is Mysql specific?

The core functionality is currently very MySQL-specific. In theory a plugin could override a bunch of methods to target Postgres, and maybe even Redis or other persistent data stores with replication and import/export functionality. It would be a lot of work though. I also made the mistake of putting "mysql" in the names of a few methods. At some point soon I'll change those to more generic names, and alias the old n…

I'd love for this to do PgSql too.

Thanks very much Evan as well for our chats ages ago (Andrew here), was happy to see you release this tool!

Post reply on HN