Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs

Post by Andys » Fri, Jun 08, 2012, 6:47 AM UTC

evanelias wrote: post #15

Andys wrote: post #14

My quick reading of this is that its suited for databases that don't change much (or at all) once the data is inserted, and not as much for apps that need to keep strong ACID compliance with guaranteed referential integrity.

We handle many thousands of write queries per second at Tumblr, and we use Jetpants to manage our entire MySQL topology, so trust me when I say the data changes quite often :) You can edit your existing posts on Tumblr, unlike on several other prominent social sites. Definitely please let me know how you got that impression though -- I'm happy to improve confusing things in the docs. As for ACID compliance: Jetpants…

Post by ralph » Fri, Jun 08, 2012, 1:15 PM UTC

_b8r0 wrote: post #22

It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space…

Post by [deleted] » Fri, Jun 08, 2012, 1:20 PM UTC

_b8r0 wrote: post #22

It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space…

Post by evanelias » Fri, Jun 08, 2012, 1:40 PM UTC

ralph wrote: post #23

_b8r0 wrote: post #22

It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space…

They're using pigz on [1] so they are compressing? I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure. socat > nc BTW, and does multicast. As for ssh, it's a shame the "no encryption" option was removed.