We Just Undid Three Months of Dev work. Here's What We Learned. • Hacker News Board

Re: We Just Undid Three Months of Dev work. Here's What We Learned.

#31

Post by acl » Thu, Oct 08, 2009, 7:24 AM UTC

mr_luc wrote: post #23

The database operations on the nested data were just taking too much processing power. Given that this is Rails, and given that it's certainly SQL involved here, I just have to ask (and I know the answer is probably "yes") -- did you try implementing nested sets? I ask because my experience has been that nested data is (with the kinds of nested data I've been handed, anyway) not a performance problem. Selects and upd…

did you try implementing nested sets?

The big thing we needed to do was a rolling archive to progressively broader timeframes. As metrics come in, we keep every single datapoint for the first 6 hours. After 6 hours, data gets rolled up into 5-minute archive. Each datapoint in the 5-minute archive then contains avg, min, max, etc for all the points that lived within that 5-minute span.

The archiving carries on through progressively broader windows as time goes on -- a 10-minute archive, 1-hr archive, etc. This progressive aggregation is the only sane approach to storing the massive amount of data we get. And, it reflects the need for higher resolution for recent events -- it's rare you need to see what happened at one exact minute 6 months ago.

It was this progressive archiving that bit us, specifically as DB performance degraded over time with lots of insertions/deletions. Nested set didn't/wouldn't help with aggregation costs and degradation from churn during the archiving process.

Hope this helps -- I'm going to try to do a more technical post on this in the future.

Re: We Just Undid Three Months of Dev work. Here's What We Learned.

#32

Post by mr_luc » Thu, Oct 08, 2009, 11:21 AM UTC

acl wrote: post #31

mr_luc wrote: post #23

The database operations on the nested data were just taking too much processing power. Given that this is Rails, and given that it's certainly SQL involved here, I just have to ask (and I know the answer is probably "yes") -- did you try implementing nested sets? I ask because my experience has been that nested data is (with the kinds of nested data I've been handed, anyway) not a performance problem. Selects and upd…

did you try implementing nested sets? The big thing we needed to do was a rolling archive to progressively broader timeframes. As metrics come in, we keep every single datapoint for the first 6 hours. After 6 hours, data gets rolled up into 5-minute archive. Each datapoint in the 5-minute archive then contains avg, min, max, etc for all the points that lived within that 5-minute span. The archiving carries on through…

Huh, that's interesting. I guess I've never worked with really "churny" data like that before.

Sure, sometimes I've had cleanup/integrity/whatever tasks that run every few minutes, but the amount of records affected has always been pretty small.

That's an interesting conundrum. See, this is why we're all messing around with Cassandra et al; sometimes, in SQL, the answer is "don't do that", because it'd be too hard to tailor the db's behavior to suit your needs. Although frankly, with a design that deletes and updates a significant percentage of records in the system on a certain schedule, I can see any number of storage solutions having trouble.

That's interesting. I'll be thinking about this at "work" today. ;) I have a bunch of comments, but they're of the half-baked "oh, what about this?" variety.

Re: We Just Undid Three Months of Dev work. Here's What We Learned.

#33

Post by zaidf » Sat, Oct 10, 2009, 1:54 PM UTC

evgen wrote: post #27

zaidf wrote: post #20

Earlier quoted context omitted.

your half-assed "iteration" has led them to decide that you don't have the chops to solve the problem While this theory sounds good, it is disproven time and again by initial half-baked versions of sites that then go on to take off. Just check the original launch of YouTube, Digg, facebook. Also, a HUGE idea coming from lean startup way is to invest very little in marketing until you have a product users like. You do…

> While this theory sounds good, it is disproven time and again by initial half-baked versions of sites that then go on to take off. Just check the original launch of YouTube, Digg, facebook. It is easy to disprove the theory if you get to cherry-pick your examples. Would you like me to list the thousands of other companies that had a couple of poorly implemented features masquerading as a "beta" that were stomped in…

Of course! I'd like to hear about them.

Btw, I don't consider using some of the most popular web2 properties as an example to be cherry picking. I'm curious to see your examples nonetheless.