Why HN was slow and how Rtm fixed it • Hacker News Board

Re: Why HN was slow and how Rtm fixed it

#21

Post by pg » Wed, Jan 19, 2011, 7:22 PM UTC

PStamatiou wrote: post #16

earle wrote: post #7

HN only supports 20 req per second???

flat files, no database

That's not the bottleneck. Essentially there's an in-memory database (known as hash tables). Stuff is lazily loaded off disk into memory, but most of the frequently needed stuff is loaded once at startup.

The bottleneck is the amount of garbage created by generating pages. IIRC there is some horrible inefficiency involving UTF-8 characters.

Re: Why HN was slow and how Rtm fixed it

#22

Post by ezalor » Wed, Jan 19, 2011, 7:22 PM UTC

gms wrote: post #4

Who's Filo? David Filo?

David Filo is the co-founder of Yahoo! if you don't know him.

Re: Why HN was slow and how Rtm fixed it

#23

Post by ximeng » Wed, Jan 19, 2011, 7:22 PM UTC

Off-topically, piecing together the years that the previous posts were made, it looks like February 4 2009 had a downvote cap of 100. It's 500 as of late last year. That suggests the annual karma inflation rate in the HN economy is sqrt(5), or ~224%.

Re: Why HN was slow and how Rtm fixed it

#24

Post by spidaman » Wed, Jan 19, 2011, 7:23 PM UTC

PStamatiou wrote: post #14

gms wrote: post #4

Who's Filo? David Filo?

That was my first thought and it would appear so. Filo was a big FreeBSDer http://zer0.org/daemons/yahoobsd.html

Filo was a total hacker, reportedly still is. At least into the late 90's, he was active on the FreeBSD mailing lists with encyclopedic knowledge of SCSI card and NIC drivers and assorted other hardware and FreeBSD stuff. He was oft found sitting on the hallway floor in the first colo I used (ISI in Mountain View), screwdrivering the chassis' of Yahoo's early servers.

Re: Why HN was slow and how Rtm fixed it

#25

Post by seanfchan » Wed, Jan 19, 2011, 7:26 PM UTC

Yes placebo, oh our brains, though its not mid day just yet :)

Re: Why HN was slow and how Rtm fixed it

#26

Post by Luyt » Wed, Jan 19, 2011, 7:28 PM UTC

klochner wrote: post #20

makmanalp wrote: post #17

Earlier quoted context omitted.

What does "web scale" mean? I see it thrown around a lot without much explanation.

http://www.youtube.com/watch?v=b2F-DItXtZs

A transcript can be found at http://mongodb-is-web-scale.com/

"Shards are the secret ingredient in the web scale sauce. They just work."

Re: Why HN was slow and how Rtm fixed it

#27

Post by ezalor » Wed, Jan 19, 2011, 7:28 PM UTC

pg wrote: post #21

PStamatiou wrote: post #16

Earlier quoted context omitted.

flat files, no database

That's not the bottleneck. Essentially there's an in-memory database (known as hash tables). Stuff is lazily loaded off disk into memory, but most of the frequently needed stuff is loaded once at startup. The bottleneck is the amount of garbage created by generating pages. IIRC there is some horrible inefficiency involving UTF-8 characters.

> That's not the bottleneck.

What is the main bottleneck of HN?

Re: Why HN was slow and how Rtm fixed it

#28

Post by pg » Wed, Jan 19, 2011, 7:32 PM UTC

ezalor wrote: post #27

pg wrote: post #21

Earlier quoted context omitted.

That's not the bottleneck. Essentially there's an in-memory database (known as hash tables). Stuff is lazily loaded off disk into memory, but most of the frequently needed stuff is loaded once at startup. The bottleneck is the amount of garbage created by generating pages. IIRC there is some horrible inefficiency involving UTF-8 characters.

> That's not the bottleneck. What is the main bottleneck of HN?

See the second paragraph.

Re: Why HN was slow and how Rtm fixed it

#29

Post by Luyt » Wed, Jan 19, 2011, 7:32 PM UTC

makmanalp wrote: post #17

Luyt wrote: post #9

"It turns out there is a hack in FreeBSD, invented by Filo, that causes the O/S not to give a new connection to the server until an entire HTTP request has arrived." I wouldn't call it a hack, but a feature ;-) # Buffer a HTTP request in the kernel # until it's completely read. apache22_http_accept_enable="yes" Is HackerNews web scale?

What does "web scale" mean? I see it thrown around a lot without much explanation.

I think it means that your web app can survive being slashdotted. So, a lightweight C10K-capable frontend, all static content on a CDN, everything heavily cached, database queries optimized to the hilt, etc... etc... Some people think that just using a NoSQL database is sufficient, but there's much more to it. Interesting talk on this stuff (was posted on HN earlier): http://ontwik.com/python/django-deployment-workshop-by-jacob...

Re: Why HN was slow and how Rtm fixed it

#30

Post by dauphin » Wed, Jan 19, 2011, 7:33 PM UTC

> It turns out there is a hack in FreeBSD, invented by Filo, that causes the O/S not to give a new connection to the server until an entire HTTP request has arrived. This might reduce the number of threads a lot, and thus improve performance; I'll give it a try today or tomorrow.

Anyone know if they're referring to "accept filters" here? FreeBSD folks can "man accf_http" if they're curious, which does prevent a request from being handed off to the application until the complete (and valid?) request has been made. Certainly not a "hack" but a feature of the OS itself.

Or they could use a proxy. All this "fuck me I'm famous" attitude is stupid.