Live data from Hacker News

How FriendFeed uses MySQL to store schema-less data

bret.appspot.com

71–80 of 92 posts

Re: How FriendFeed uses MySQL to store schema-less data

#71
post #32
post #25

Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features th…

Hi, is there any chance you could have a look at this comment: http://news.ycombinator.com/item?id=497070 and let me know how you resolve this problem in practice? (Or correct my understanding if there is no problem really).

Increasing the number of shards is similar to changing the backend infrastructure, but simpler. Downtime obviously isn't acceptable, so when switching from one system to the next, we have a period during which we write to both so that it is safe to read from either one. Other schemes could be used for resharding, but this is simple enough and also works for other changes, and in practice we've changed the schema more often than the number of shards.

Also, keep in mind that it is ok to have more shards than you really need (multiple shards can run on the same machine, for example), so resharding needn't be a common operation.

Re: How FriendFeed uses MySQL to store schema-less data

#72
post #22

Earlier quoted context omitted.

Yeah. They actually explained that in the article too - the shard number = user_id % num_of_shards. So user 1 is on DB1, user 2 is on DB3, etc. If they have 10 shards, user 11 starts back on DB1 etc.

I don't understand how that scheme can work, since changing the number of shards changes the location of most users. e.g. we have 4 shards, so user 5 is on shard1. If we go to 6 shards, user5 is now on shard5. I guess it works with downtime to move the users, or another layer of indirection, where the newly created shards can "point back" to existing shards, but otherwise I don't see it. My understanding of sharding…

You make a large number of virtual shards. Each virtual shard is mapped to some machine. When you add a new machine, you move some shards to the new real machine.

Presumably consistent hashing is also helpful here.

Re: How FriendFeed uses MySQL to store schema-less data

#73
This is great. I sketched out a similar design for the Delicious backend, but the team that built it went in a completely different and stupid direction. Oh well.

Mine also had:

- some way to do server locality (I don't think FF needs this)

- some way to do table locality (I don't think FF needs this either)

- per blob versioning and type

- a virtual shard layer, so new machines can be added inexpensively

Sorry for any inadvertent whining, incidentally.

Re: How FriendFeed uses MySQL to store schema-less data

#74
post #24
post #14

Earlier quoted context omitted.

It's really nice for not having to bring down the site for schema upgrades. All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.

Actually we're on postgres... It just makes me nervous when the migration is taking upwards of 10 minutes to complete since some of our tables have millions of rows and lots of concurrent read/writes are happening.

I don't think executing long-running DDL with concurrent read/writes is that bad, especially if the DDL is additive and doesn't lock the whole table.

Concurrency - isn't that what a DB is supposed to do well?

Re: How FriendFeed uses MySQL to store schema-less data

#75
post #30

Earlier quoted context omitted.

While your approach to do this in JSON is cool, I think you have overlooked the 'direct' solution to do this in a RDBMS - with a linked list. Here is a quick suggestion (works in Postgres): create table l ( id char primary key references l(prev) deferrable initially deferred, prev char unique not null references l(id) deferrable initially deferred, mydata text not null ); then I populate the table with your example i…

Aha! I did forget the linked list approach. So essentially, each item on the list stores what is before (or) after it. I'm assuming it's an arbitrary choice that you're using "prev" instead of "next," correct? The use of deferred, I've never heard of, but it makes perfect sense in this case. Unless, of course, you want to insert the record first and then modify the update to exclude the item you just inserted. Right…

> I'm assuming it's an arbitrary choice that you're using "prev" instead of "next," correct?

yes, in effect it's a doubly linked list (circularly doubly linked), so you could remember the id of the first element of the list and then traverse in any order as long as this id does not reappear.

> Unless, of course, you want to insert the record first and then modify the update to exclude the item you just inserted.

yes - in this case you would get a violation of the unique constraint:

    begin;
    insert into l (id, prev, mydata) values ('G', 'C', 'data for G');
    update l set prev='G' where prev='C';
    commit;
this would violate the unique constraint for prev, because after the insert (but before the update!), both the new element and the element not yet updated have prev set to 'C' - the transaction will then be rolled back.

In standard SQL this would be possible because it allows to declare unique constraints (and I think other constraints, such as check clauses) as deferrable, too - PostgreSQL doesn't implement this, it allows deferrable only for foreign key constraints. But usually it's no problem to order the commands in a way that only foreign key constraints get violated during a transaction.

> Right now, I'm using MySQL.. I only have 4 tables, and product is not launched. Would you advise switching to Postgres?

as an entrepreneur, you should probably do what's best for your customers - and they will likely not care which RDBMS you use:-) Perhaps you could play around a little bit with PostgreSQL on the side and (perhaps) make the switch once you are comfortable with it. And keep your JSON-based lists - if the system works, why bother with a rewrite / schema change (for now?).

Considering momentum, I think PostgreSQL is gaining steam while MySQL is losing momentum (some key developers left after the aquisition by Sun) - of course, that's my subjective impression.

Technically, of course I think that PostgreSQL is better - here is a good comparison: http://www.wikivs.com/wiki/MySQL_vs_PostgreSQL for amusement, read the discussion here: http://www.reddit.com/r/programming/comments/764fp/mysql_vs_...

Re: How FriendFeed uses MySQL to store schema-less data

#76
post #34
post #21

Earlier quoted context omitted.

No, there are no silver bullets, but it seems to me that people reach for Mysql a bit too quickly, without considering the pros and cons. And while it's improving, Mysql has had many frustrating things in the past... to me it's always seemed like a "worse is better" kind of thing. Sure, it's "fast", but at what cost? Once you go to InnoDB, you lose that speed advantage. One thing that's not a tech tradeoff, and is ge…

Actually, Postgres doesn't have any great out-of-the-box solution for partitioning the database across machines. The usual suggestion is Slony, but that is no where near as robust and widely deployed as MySQL replication. The GPL licence for MySQL isn't really a problem for webapps anyway. OTOH, Postgres does somewhat better than MySQL on a single box with multiple cores (it's fairly linear up to 8 CPU, which is much…

Postgres does somewhat better than MySQL on a single box with multiple cores ... mainly because of the work Sun put into scaling it before they bought MySQL.

That's not really the case. Sun has some some benchmarking, but most of the work on Postgres SMP performance was done by others (mostly Tom Lane, who works for Red Hat, and various EnterpriseDB employees).

Re: How FriendFeed uses MySQL to store schema-less data

#77
post #58

Earlier quoted context omitted.

Correct, you don't split shards across machines. Each machine hosts x number of shards.

OK, but # of shards is fixed "for ever" under the modulus scheme? You pick it once, when you first shard and then you're looking at downtime to adjust it? In order to split across multiple dbs, you're looking at creating say 100/1000 dbs in our initial split (when you've got maybe 2-3 machines). And that number then caps the number of machines you can scale to without adding another layer (sharding-shards) or having…

> You pick it once, when you first shard and then you're looking at downtime to adjust it?

Yes, but you say that like it's a bad thing, it's not. If you're ever forced to reshard, that means you grew beyond what you ever hoped... hurray, awesome, nice problem to have. The reality is however, 99% chance that'll never happen.

Like 1 in a 1000 people ever run into a scaling problem of this magnitude but if you read the blogs you'd come away thinking scaling issues that require sharding are common and everyone needs this stuff, but they aren't, and they don't.

Re: How FriendFeed uses MySQL to store schema-less data

#79
post #35
post #33

Earlier quoted context omitted.

How do hash joins help with sharding? Surely once you go above a certain number of writes per second you're going to need to have more than one writable database server, at which point you need to start partitioning and better join algorithms aren't going to do anything to help.

If you can't hash join then you can't join over large datasets anyway, so sharding costs you nothing in that respect. Right now, using off the shelf kit and doing nothing particularly clever, running a major commercial RDBMS you could do 10,000 commits/sec and handle 100T of data on a single instance. Sure it would cost you a pretty penny, but the thing is, unless running a database is the one competitive advantage y…

I'm not too sure if you are arguing for or against the commercial RDBMS.

To me, that sounds like a pretty good argument against it - the great thing about MySQL/Postgres+Sharding is that it scales down, as well as up. You can start out with a single server, then gradually add in extra servers as you need them. With the "single big DB server model" it doesn't work like that - you have to make a pretty decent investment early on in the software licence, build your software to use it, and then the pricing isn't linear, either.

Plus, the backup/redundancy thing sucks too - with sharded MySQL you need a couple of spare cheap servers, but with Oracle etc you need to pay twice for the licence and a second server.

OTOH, in a corporate environment it's much easier to predict your usage and the pricing is easier to justify. Also your priority is safety+justifiability first rather than price or even price/performance.

Re: How FriendFeed uses MySQL to store schema-less data

#80
post #28

Earlier quoted context omitted.

Indeed. This is not a RDBMS problem. Its a MySQL problem and the correct solution is to use a mature DB.

Even in Oracle, we've had to schedule downtime for a single column schema change and population for 45 minutes. I would call Oracle "mature." Note: I wasn't involved in doing the update, so it's possible there was a better way to do it. The table is used by about 50 different applications.

Almost all major oracle operations can be done online ... Index rebuilds, table reorgs etc. Depends how much hardware you have how much it affects things ...
Post reply on HN