Reading this makes me realize how naive my DB setup is for my startup.
How FriendFeed uses MySQL to store schema-less data
61–70 of 92 posts
Re: How FriendFeed uses MySQL to store schema-less data
#62Earlier quoted context omitted.
I don't understand how that scheme can work, since changing the number of shards changes the location of most users. e.g. we have 4 shards, so user 5 is on shard1. If we go to 6 shards, user5 is now on shard5. I guess it works with downtime to move the users, or another layer of indirection, where the newly created shards can "point back" to existing shards, but otherwise I don't see it. My understanding of sharding…
I don't see how this scheme can scale simply for the reason that there's no built in balancer. What's to stop shardN from becoming overwhelmed when all the power users end up there, while shardN-1 has no activity?
flickr deals with that problem by manually moving power users if necessary. They have a tool that lets them do that. See: http://highscalability.com/how-flickr-handles-moving-you-ano...
In practice it's going to be very rare that you'll overwhelm a single shard and adding an extra layer to point to where people actually are is quite simple and fast. You can use a simple cache key (read-through cache of course) that, if it exists means the user is on a specific shard, overriding the default algorithmic pick.
Re: How FriendFeed uses MySQL to store schema-less data
#63Earlier quoted context omitted.
Correct, you don't split shards across machines. Each machine hosts x number of shards.
OK, but # of shards is fixed "for ever" under the modulus scheme? You pick it once, when you first shard and then you're looking at downtime to adjust it? In order to split across multiple dbs, you're looking at creating say 100/1000 dbs in our initial split (when you've got maybe 2-3 machines). And that number then caps the number of machines you can scale to without adding another layer (sharding-shards) or having…
Yes. In the simple modulo scheme, the number of shards is fixed. Unless you want to re-shard the entire thing. (AFAIK)
Re: How FriendFeed uses MySQL to store schema-less data
#64Earlier quoted context omitted.
I've been evaluating databases recently. It doesn't matter if the site is not well-known in the US as long as you can post traffic numbers. TokyoTyrant is Japanese, after all. Also these help: * able to handle large (>200GB) datasets * client libraries for top N languages * easy way to write client libs (eg simple protocol, a C library, etc) * connection pooling and/or cheap connections * easy to install on Mac, Linu…
Agreed on most of the points... about MySQL, I developed Redis just because MySQL does not scale enough with given kind of datasets :)
Scale enough in which way? Read traffic? Write traffic? Locking? If you use MySQL as a btree it is pretty fast and consistent. I gather from this post that a lot of companies started in the last 18 months are doing this. It's a hack but a remarkably useful one. Good luck! I will keep my eye on redis.
Re: How FriendFeed uses MySQL to store schema-less data
#65Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features th…
I tried something like this before but thought it was a "naive way to do things".
I love the advice that sometimes doing what may seem like a naive thing is good enough.
Re: How FriendFeed uses MySQL to store schema-less data
#66Earlier quoted context omitted.
I just recently wrapped up using a similar "store serialized data" set-up myself. In my example, I'm allowing the user insert/remove/reorder items on a list. This type of operation is pain to do with SQL operations. If you have a list, A,B,C,D,E,F and you want to insert G before C, then you either have to: delete all, then insert A,B,G,C,D,E,F. set "sortnumber" on G to "3", and increment sortnumber on all >= 3. set "…
While your approach to do this in JSON is cool, I think you have overlooked the 'direct' solution to do this in a RDBMS - with a linked list. Here is a quick suggestion (works in Postgres): create table l ( id char primary key references l(prev) deferrable initially deferred, prev char unique not null references l(id) deferrable initially deferred, mydata text not null ); then I populate the table with your example i…
Aha! I did forget the linked list approach. So essentially, each item on the list stores what is before (or) after it. I'm assuming it's an arbitrary choice that you're using "prev" instead of "next," correct?
The use of deferred, I've never heard of, but it makes perfect sense in this case. Unless, of course, you want to insert the record first and then modify the update to exclude the item you just inserted.
Right now, I'm using MySQL.. I only have 4 tables, and product is not launched. Would you advise switching to Postgres?
Re: How FriendFeed uses MySQL to store schema-less data
#67"However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs" Ok, just some hour ago I released the beta-3 of Redis ( http://code.google.com/p/redis/ if you care) and I'm near to feature-freeze with exactly with this goal. To make it rock solid (I'm going to use it in my start…
How is redis different from TokyoTyrant?
Has anyone here got TokyoTyrant working under Windows? I found a static library binary for Windows, but the header files need to compile are either not there, or the code references some bizarre POSIX stuff which isn't easy to hack into working under Windows.
Re: How FriendFeed uses MySQL to store schema-less data
#68I love this key-value store trend. Looking at CouchDB in particular has made databases are fun again.
Re: How FriendFeed uses MySQL to store schema-less data
#69Earlier quoted context omitted.
Agreed on most of the points... about MySQL, I developed Redis just because MySQL does not scale enough with given kind of datasets :)
Scale enough in which way? Read traffic? Write traffic? Locking? If you use MySQL as a btree it is pretty fast and consistent. I gather from this post that a lot of companies started in the last 18 months are doing this. It's a hack but a remarkably useful one. Good luck! I will keep my eye on redis.
Probably it just depends on the dataset. The issue we have with our service (http://lloogg.com) is that you need to take the last N items of logs for every site. To ask for the latest M items should be fast. To push a new log line on the list should be fast. Every kind of MySQL configuration we tried was unable to reach the 10k writes/second we reach with Redis. Obviously. Even when you use MySQL as a btree implementation you get a lot of overhead. Starting from the protocol and the format of the statements, for example.
The idea to encode things with json or other formats in a blog text is just a ugly hack. People are using this because they are desperate, not because is good computer science. They started with mysql, know mysql, hacked with mysql. Clearly will try to fix their site with MySQL.
The json+blob can work as long as the data that's stored in this fields is trivial to serialize-deserialize. What about having a 10000 elements list in every blob and at every page view you need to append an element?
So: great hack, you found a way to work with the tools you have, but this does not mean in any way that fast key-value persistent DBs don't have something to say into the web-scale theater.
Re: How FriendFeed uses MySQL to store schema-less data
#70Earlier quoted context omitted.
We don't do any joins in MySQL. We query the indexes in all of the shards in parallel to get a list of entity IDs and then query the entities tables in all of the shards in paralel to get the entity bodies as a second operation.
But would you, if MySQL could do hash joins?
[deleted]