Live data from Hacker News

Why does digg need so many servers?

twitter.com

61–65 of 65 posts

Re: Why does digg need so many servers?

#61
post #34

Earlier quoted context omitted.

For someone who took $ x million "off the table," during those funding rounds the hypocrisy is impressive.

Since when did "experience" == "hypocrisy"?

Huh? Kevin sold several million dollars worth of stock, enabled by raising more money than needed, and gave up more control to do it. Now he advocates that others raise as little money as possible. How is that "experience?"

Re: Why does digg need so many servers?

#62
post #57
post #42

The difference is that digg has network information (IE: Followers). Stack overflow doesn't. Think through the ramifications of each of these page flows: 1. Give me this question and everybody who commented on it. 2. Give me every post that this user is following has made over the last n days. That's why.

That would increase the load on the database. Why does that increase the load on the webservers?

It depends how you approach the problem. I'm guessing here, from my own experience implementing social graph features, but here goes:

There are a couple of ways you can go about this, the first is the database: Join the network of people against the land of content and bring it back. This doesn't work (as an aside this is what people mean when they say web scale, it has nothing to do with web traffic, it's social graphs) your database will cry. All though not at first, in development it works fine, and you feel fine, and for a while you're ok, but you start growing....

Another way you can go about it is by denormalizing. In this world you store a pointer to each content item for each user. So anytime I do something all the people [following|watch|connected|friended] to me get a record indicating I did this. This works, but now you have lots of data (lots and lots of data!) spread all crazy around. You need some kind of system to push that data out to everybody. It's those last two that drive up your hardware usage, it's not necessarily web boxes, but it's boxes in the background broadcasting the events out to the world, and the datastores to hold it all. Depending on how your web code works you could also have a lot of overhead on the webservers putting all that stuff together.

My experience here comes from building the social features into toolbox.com. A good example is this page http://it.toolbox.com/people/george_krautzel/posts-connectio... That's all the posts from users connected to our CEO (all 750k of them). Getting that to return in near real time is super fun (and you can probably tell that I went down the DB join path before it all fell apart).

Re: Why does digg need so many servers?

#63
post #47
post #26

Earlier quoted context omitted.

Absolutely. Every time someone trots out "And Reddit only has 7 guys", I think to myself that they're innovating as if they had 3. Say what you will about Digg, but at least they rolled out new software.

Say what you will about Digg, but at least they rolled out new software. Digg is a discussion site, not an operating system -- what is the value in "rolling out new software"? Digg's value is in the quality of the discussion and community; "innovation" that doesn't improve that is pointless. Look at HN: it remains nearly as minimalist as always, and mostly receives occasional tweaks and performance improvements.

Sure, you can argue that they would have been better off not rolling out new software, or that they don't need new features. I'm just saying that the answer to the (sometimes implicit) question of "why does Digg have so many more people than Reddit" is that some of them are designing features and writing code, and there doesn't seem to be much of that going on at Reddit.

Personally, I'm in general agreement with you. And you can rebase the whole argument: take MetaFilter, with 4 people on the payroll (and not all technical). What is everyone at Reddit doing?

Re: Why does digg need so many servers?

#64
I mentioned this to Spolsky already but depending how you want to count servers (web servers or all front facing servers or all servers in your infrastructure?), we have 10-20 servers servicing 350 million page views per month. That is of course not counting all the ajax requests that we also service, which don't count as page views. So we service 17-35 million page views per server, compared to StackExchange's 12 million page views per server.

We use PHP and Apache, so the gloating about ASP being way more efficient than PHP seems to be unfounded.

Re: Why does digg need so many servers?

#65
post #38

Earlier quoted context omitted.

Not to mention a basically non-existent cachability profile, and an order of magnitude more functionality.

I'm assuming you mean Facebook has more functionality and no cachebility. That might be true but I wonder how expensive getting a Digg discussion(with 100-200 posts) page compared to say a 5 answer SO question or even just a simple Facebook profile.

As I understand it, a Facebook profile is anything but simple.
Post reply on HN