Live data from Hacker News

How FriendFeed uses MySQL to store schema-less data

bret.appspot.com

21–30 of 92 posts

Re: How FriendFeed uses MySQL to store schema-less data

#21
post #20
post #9

Earlier quoted context omitted.

Pretty much it's MySQL only. Most of the MySQL posts here are actually about clever ways to do things most DBAs and developers have taken for granted for years. Do you wonder why no-one seems to be writing blog posts about "sharding" Sybase or DB2 or Oracle or Postgres...?

Actually, they do, e.g. http://highscalability.com/skype-plans-postgresql-scale-1-bi... MySQL is popular with startups though, so it's not surprising that there's a lot written about it. It's true that MySQL has some lame limitations, but I don't believe that there are any silver bullets out there. Google tried to switch their ads system from MySQL to a "real" database once, and it was basically a disaster and had to…

No, there are no silver bullets, but it seems to me that people reach for Mysql a bit too quickly, without considering the pros and cons. And while it's improving, Mysql has had many frustrating things in the past... to me it's always seemed like a "worse is better" kind of thing. Sure, it's "fast", but at what cost? Once you go to InnoDB, you lose that speed advantage.

One thing that's not a tech tradeoff, and is generally a Postgres win, is the BSD style licensing. You can take Postgres and do whatever you want with it with no worries.

> Google tried to switch their ads system from MySQL to a "real" database once,

I hope the part that actually handles money has been fiddled with by Google to be robust.

Re: How FriendFeed uses MySQL to store schema-less data

#22

Earlier quoted context omitted.

When they say it's sharded on "user_id" what they mean is that's the field that decides which database the record is stored. It might go something like: if the user starts with 0-8, store in DB1, otherwise, store in DB2. This is up to their Datastore controller to decide how to hash based on the user_id and the number of databases.

Yeah. They actually explained that in the article too - the shard number = user_id % num_of_shards. So user 1 is on DB1, user 2 is on DB3, etc. If they have 10 shards, user 11 starts back on DB1 etc.

I don't understand how that scheme can work, since changing the number of shards changes the location of most users.

e.g. we have 4 shards, so user 5 is on shard1. If we go to 6 shards, user5 is now on shard5.

I guess it works with downtime to move the users, or another layer of indirection, where the newly created shards can "point back" to existing shards, but otherwise I don't see it.

My understanding of sharding is that it's best to have a global lookup table user->shard. That's not a huge amount of data, even for millions of users.

Anyone care to educate me as to how they get user%num_shards working in practice?

Re: How FriendFeed uses MySQL to store schema-less data

#23
post #14
post #8

That's reassuring, I thought I was the only crazy fool who was storing json objects in database columns =). We do something similar for some of our data models at thesixtyone.com. It's really nice for not having to bring down the site for schema upgrades.

It's really nice for not having to bring down the site for schema upgrades. All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.

Indeed. This is not a RDBMS problem. Its a MySQL problem and the correct solution is to use a mature DB.

Re: How FriendFeed uses MySQL to store schema-less data

#24
post #14
post #8

That's reassuring, I thought I was the only crazy fool who was storing json objects in database columns =). We do something similar for some of our data models at thesixtyone.com. It's really nice for not having to bring down the site for schema upgrades.

It's really nice for not having to bring down the site for schema upgrades. All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.

Actually we're on postgres... It just makes me nervous when the migration is taking upwards of 10 minutes to complete since some of our tables have millions of rows and lots of concurrent read/writes are happening.

Re: How FriendFeed uses MySQL to store schema-less data

#25
Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features that would have been problematic in the old system.

This experience reinforces my belief that it's better to be quick than brilliant. If we had wasted a lot of time trying to build some really smart, super-scalable system right from the start, it would have inevitably been over-optimized for the wrong things, since the product and requirements have changed quite a bit since launch. By continually rebuilding the system, it stays relatively simple and close to our actual needs (plus it incorporates everything we learned from the previous iterations).

Re: How FriendFeed uses MySQL to store schema-less data

#26
Excellent article. I work in an environment where we have serious vendor lock-in and the people in charge of infrastructure have a real fear of change. I have free reign to implement in any improvements in our performance and code so long as I don't make things difficult for them.

Thus, the idea of a MySQL cache system is something I've been planning to look at for a while as it plays with the existing infrastructure. This (seemingly proven) system will make things a lot easier, so thanks for sharing.

Re: How FriendFeed uses MySQL to store schema-less data

#27

This is really interesting. We did something similar for GameClay. I stored game properties as JSON-encoded dicts stored in MogileFS, then had a "regular" MySQL table that would point to the MogileFS key for the file, then the Python code would just read it out, use a JSON library to parse it, and manipulate it as a Python object. We had normal MySQL indexes on all the game metadata that appeared in the UI, so if you…

I just recently wrapped up using a similar "store serialized data" set-up myself. In my example, I'm allowing the user insert/remove/reorder items on a list. This type of operation is pain to do with SQL operations. If you have a list, A,B,C,D,E,F and you want to insert G before C, then you either have to: delete all, then insert A,B,G,C,D,E,F. set "sortnumber" on G to "3", and increment sortnumber on all >= 3. set "…

> It takes a fraction of the amount of time to do native data structure stuff than it does to touch the DB several times.

And app processes are much easier to scale than a database. Even in "slow" languages (Ruby in my case) array operations are extremely fast, definitely "fast enough".

Re: How FriendFeed uses MySQL to store schema-less data

#28
post #14

Earlier quoted context omitted.

It's really nice for not having to bring down the site for schema upgrades. All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.

Indeed. This is not a RDBMS problem. Its a MySQL problem and the correct solution is to use a mature DB.

Even in Oracle, we've had to schedule downtime for a single column schema change and population for 45 minutes. I would call Oracle "mature."

Note: I wasn't involved in doing the update, so it's possible there was a better way to do it. The table is used by about 50 different applications.

Re: How FriendFeed uses MySQL to store schema-less data

#29
post #22

Earlier quoted context omitted.

Yeah. They actually explained that in the article too - the shard number = user_id % num_of_shards. So user 1 is on DB1, user 2 is on DB3, etc. If they have 10 shards, user 11 starts back on DB1 etc.

I don't understand how that scheme can work, since changing the number of shards changes the location of most users. e.g. we have 4 shards, so user 5 is on shard1. If we go to 6 shards, user5 is now on shard5. I guess it works with downtime to move the users, or another layer of indirection, where the newly created shards can "point back" to existing shards, but otherwise I don't see it. My understanding of sharding…

I don't see how this scheme can scale simply for the reason that there's no built in balancer. What's to stop shardN from becoming overwhelmed when all the power users end up there, while shardN-1 has no activity?

Re: How FriendFeed uses MySQL to store schema-less data

#30

This is really interesting. We did something similar for GameClay. I stored game properties as JSON-encoded dicts stored in MogileFS, then had a "regular" MySQL table that would point to the MogileFS key for the file, then the Python code would just read it out, use a JSON library to parse it, and manipulate it as a Python object. We had normal MySQL indexes on all the game metadata that appeared in the UI, so if you…

I just recently wrapped up using a similar "store serialized data" set-up myself. In my example, I'm allowing the user insert/remove/reorder items on a list. This type of operation is pain to do with SQL operations. If you have a list, A,B,C,D,E,F and you want to insert G before C, then you either have to: delete all, then insert A,B,G,C,D,E,F. set "sortnumber" on G to "3", and increment sortnumber on all >= 3. set "…

While your approach to do this in JSON is cool, I think you have overlooked the 'direct' solution to do this in a RDBMS - with a linked list. Here is a quick suggestion (works in Postgres):

    create table l (
           id char primary key references l(prev) deferrable initially deferred,
           prev char unique not null references l(id) deferrable initially deferred,
           mydata text not null
    );
then I populate the table with your example items:

    insert into l (id, prev, mydata) values ('A', 'F', 'dA'),
                                            ('B', 'A', 'dB'),
                                            ('C', 'B', 'dC'),
                                            ('D', 'C', 'dD'),
                                            ('E', 'D', 'dE'),
                                            ('F', 'E', 'dF');
let's see how that looks like:

    test=# select * from l;
    select * from l;
     id | prev | mydata 
    ----+------+--------
     A  | F    | dA
     B  | A    | dB
     C  | B    | dC
     D  | C    | dD
     E  | D    | dE
     F  | E    | dF
     (6 rows)
to insert a new item into the list, you would do:

    begin;
    update l set prev='G' where prev='C';
    insert into l (id, prev, mydata) values ('G', 'C', 'data for G');
    commit;
so that's one update, one insert for an insertion into the list. Note that the two commands have to be in one transaction, because inside the transaction the foreign key constraint is violated (as allowed by the deferrable initially deferred modifier).

Let's inspect our list again:

    test=# select * from l;
    select * from l;
     id | prev |   mydata   
    ----+------+------------
     A  | F    | dA
     B  | A    | dB
     C  | B    | dC
     E  | D    | dE
     F  | E    | dF
     D  | G    | dD
     G  | C    | data for G
so the predecessor of G is C, and the predecessor of D is G, like specified.

Of course, you loose the ability to sort with 'order by', but that's no big deal: you know the predecessor and successor of each item, so it's easy to traverse the list in either order. This could be done on the client side [probably the best solution in your case], in the application code, or inside the database with a stored procedure or with a recursive query (coming in PostgreSQL 8.4), in Oracle it could probably be done with 'connect by'.

In reality, you would of course choose other datatypes for id and prev (probably integer), but I wanted to translate your example as literally as possible. Another problem that's easily solved: how do I get all elements of one list? Solution: Either give me one 'starting element' and the list is traversed and returned. Or introduce a listId attribute and select by that, which is probably faster but without sort order.

Post reply on HN