Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How FriendFeed uses MySQL to store schema-less data

bret.appspot.com

Post reply on HN
Print view View original

41–50 of 92 posts

Re: How FriendFeed uses MySQL to store schema-less data

#41

Post by simonw »

gaius wrote: post #35
simonw wrote: post #33

Earlier quoted context omitted.

How do hash joins help with sharding? Surely once you go above a certain number of writes per second you're going to need to have more than one writable database server, at which point you need to start partitioning and better join algorithms aren't going to do anything to help.

If you can't hash join then you can't join over large datasets anyway, so sharding costs you nothing in that respect. Right now, using off the shelf kit and doing nothing particularly clever, running a major commercial RDBMS you could do 10,000 commits/sec and handle 100T of data on a single instance. Sure it would cost you a pretty penny, but the thing is, unless running a database is the one competitive advantage y…

Which commercial database? Any idea which web scale applications are using it, and why the other ones aren't?

Re: How FriendFeed uses MySQL to store schema-less data

#42

Post by [deleted] »

paul wrote: post #25

Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features th…

[deleted]

Re: How FriendFeed uses MySQL to store schema-less data

#43

Post by abstractbill »

paul wrote: post #25

Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features th…

This experience reinforces my belief that it's better to be quick than brilliant...

I've come to the same conclusion during the past year and a half of working on Justin.TV. If I was writing a list of pieces of startup advice, this would be #1.

Re: How FriendFeed uses MySQL to store schema-less data

#44

Post by mbreese »

jbert wrote: post #40
gnaritas wrote: post #39

Earlier quoted context omitted.

You don't change the number of shards, you change the number of machines and re-balance the shards across them. Shard != machine. 12 shards on 1 machine 6 shards on 2 machines 4 shards on 3 machines 3 shards on 4 machines 2 shards on 6 machines And on and on...

Ah, OK. So you start with enough shards "for ever" and not change the number? Edit: no wait. If you can split a shard across multiple machines, what's the benefit of having more than 1 shard? Why not have 1 shard split across 1000 machines?

The above table wasn't too clear, but my understanding is that a shard exists on only a single machine, but that a single machine could host multiple shards.

Re: How FriendFeed uses MySQL to store schema-less data

#45

Post by ryanwaggoner »

nostrademons wrote: post #4

This is really interesting. We did something similar for GameClay. I stored game properties as JSON-encoded dicts stored in MogileFS, then had a "regular" MySQL table that would point to the MogileFS key for the file, then the Python code would just read it out, use a JSON library to parse it, and manipulate it as a Python object. We had normal MySQL indexes on all the game metadata that appeared in the UI, so if you…

This is really interesting...could you do the same thing using S3 instead of MogileFS? The advantage would be cost and simple scalability, especially if you're running on EC2 so you don't have to pay for all the back and forth transfer from EC2 to S3. Concerns might be latency issues and whether this would scale to billions of objects. Other than that, it would seem like you could run a pretty huge site off of just a couple beefy servers.

Re: How FriendFeed uses MySQL to store schema-less data

#46

Post by bmj »

gaius wrote: post #14
hsiung wrote: post #8

That's reassuring, I thought I was the only crazy fool who was storing json objects in database columns =). We do something similar for some of our data models at thesixtyone.com. It's really nice for not having to bring down the site for schema upgrades.

It's really nice for not having to bring down the site for schema upgrades. All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.

I find this interesting because for a previous project, we had to manage separate database instances for each client (regulations). Even though we had a database management tool that allowed us to push schema changes out to the various DBs automatically, it would be better to not have to push those changes out at all.

Re: How FriendFeed uses MySQL to store schema-less data

#47

Post by antirez »

"However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs"

Ok, just some hour ago I released the beta-3 of Redis (http://code.google.com/p/redis/ if you care) and I'm near to feature-freeze with exactly with this goal. To make it rock solid (I'm going to use it in my startup's web stuff with a lot of users/month, so I care about stability).

The question is: what's in your opinion the right path to make a system like Redis stable and reliable for the real world usage? What to publish on the site in order to inspire a good feeling about stability? Thanks

Re: How FriendFeed uses MySQL to store schema-less data

#48

Post by pxlpshr »

paul wrote: post #25

Another interesting detail is that this is roughly the 4th iteration on the FriendFeed backend since we launched 17 months ago. If you look at the the graphs at the bottom of Bret's post, you can see that our previous system was about to die -- average pageview latency had increased from about 135ms to 260ms in less than a month! (not a good trend) This new design also accommodates some important upcoming features th…

After reflecting of my own experience with a failed B2C product, I definitely agree with the statement in your second paragraph. We spent a significant amount of time worrying about 'scaling' on a lot of fronts, not just technical, but the highest CCU we ever hit was a couple hundred users over the course of a year and a half.

From a design standpoint, I think its perfectly acceptable to keep said objectives a high priority... but from an execution standpoint, its more important to be as nimble and flexible as possible. Re-writes shouldn't be feared too much, they take less and less time if the team is applying what they've learned.

Re: How FriendFeed uses MySQL to store schema-less data

#49

Post by ropiku »

jbert wrote: post #22
nostrademons wrote: post #12

Earlier quoted context omitted.

Yeah. They actually explained that in the article too - the shard number = user_id % num_of_shards. So user 1 is on DB1, user 2 is on DB3, etc. If they have 10 shards, user 11 starts back on DB1 etc.

I don't understand how that scheme can work, since changing the number of shards changes the location of most users. e.g. we have 4 shards, so user 5 is on shard1. If we go to 6 shards, user5 is now on shard5. I guess it works with downtime to move the users, or another layer of indirection, where the newly created shards can "point back" to existing shards, but otherwise I don't see it. My understanding of sharding…

Instead of using normal hashing you can use consistent hashing so you don't need to move that many users. Here's a post explaining it: http://www.spiteful.com/2008/03/17/programmers-toolbox-part-...

Re: How FriendFeed uses MySQL to store schema-less data

#50

Post by davidw »

nl wrote: post #34
davidw wrote: post #21

Earlier quoted context omitted.

No, there are no silver bullets, but it seems to me that people reach for Mysql a bit too quickly, without considering the pros and cons. And while it's improving, Mysql has had many frustrating things in the past... to me it's always seemed like a "worse is better" kind of thing. Sure, it's "fast", but at what cost? Once you go to InnoDB, you lose that speed advantage. One thing that's not a tech tradeoff, and is ge…

Actually, Postgres doesn't have any great out-of-the-box solution for partitioning the database across machines. The usual suggestion is Slony, but that is no where near as robust and widely deployed as MySQL replication. The GPL licence for MySQL isn't really a problem for webapps anyway. OTOH, Postgres does somewhat better than MySQL on a single box with multiple cores (it's fairly linear up to 8 CPU, which is much…

It's not as robust? Do you have a source for that? You're doubtless right about it not being as widely used.

I've always preferred the fact that Postgres tried to do things correctly. Most recently, I bumped into this with Mysql, and it reminded me why I get irritated when I use it:

http://journal.dedasys.com/2008/11/11/another-mysql-doesnt-d...

I've often bumped into things like that that just irk me.

Post reply on HN