Making a site that can handle #1 on Hacker News • Hacker News Board

Re: Making a site that can handle #1 on Hacker News

#21

Post by saurik » Mon, Apr 29, 2013, 4:05 AM UTC

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly benign decay. How could you possibly have been dealing with 287 concurrent users?

My website (saurik.com) is seriously written in JavaScript. I was doing this long before node was popular, and so it is designed "horribly sub-optimally": it is using Rhino, which is not known for speed. I use XSL/T as a templating engine to build the page layouts, which is also not known for speed. Every request is synchronously logged to a database. I get over 50k HTML pageviews a day, most for one recent article which I posted a few weeks ago: when I posted it, I was getting well over 3k pageviews per hour.

I do not do any caching: I generate each page dynamically every time it is accessed. I seriously dynamically generate the CSS every request (there are some variables). Even with 3k HTML pageviews per hour, that's less than one complex request per second. How does one even build a website that can't handle that load? That is what I'd seriously be interested in seeing: not "how do I handle being #1 on Hacker News", but "why is it that so many websites are unable to handle two requests per minute".

Re: Making a site that can handle #1 on Hacker News

#22

Post by tahoecoder » Mon, Apr 29, 2013, 4:05 AM UTC

jacques_chester wrote: post #19

Without wanting to seem unnecessarily rude, it's not that hard to survive #1 at HN. I had a blog post submitted by someone else hit #1 for two days. It was worth about 25k visits. I've had other stuff do 100k visits in a day on the same system during a natural disaster; that little network is basically idling at what used to be vanity numbers (millions of views per month, guys, time to list on NASDAQ!) If you're on W…

A lot of people on HN are still learning things like this. I understand that a post like this doesn't add any value to someone like you who probably knows this stuff inside and out, but I'm grateful for posts that give me insight into new tools that can help me. This isn't just a community for people with 10 years of programming experience.

Should I not try to help out the community with blog posts about my experience? Should we just cater to the experts?

Re: Making a site that can handle #1 on Hacker News

#23

Post by jacques_chester » Mon, Apr 29, 2013, 4:11 AM UTC

tahoecoder wrote: post #22

jacques_chester wrote: post #19

Without wanting to seem unnecessarily rude, it's not that hard to survive #1 at HN. I had a blog post submitted by someone else hit #1 for two days. It was worth about 25k visits. I've had other stuff do 100k visits in a day on the same system during a natural disaster; that little network is basically idling at what used to be vanity numbers (millions of views per month, guys, time to list on NASDAQ!) If you're on W…

A lot of people on HN are still learning things like this. I understand that a post like this doesn't add any value to someone like you who probably knows this stuff inside and out, but I'm grateful for posts that give me insight into new tools that can help me. This isn't just a community for people with 10 years of programming experience. Should I not try to help out the community with blog posts about my experienc…

I feel like we're going to sail off into old/new user debate territory. "It's been done before" / "I'm new and so it's novel to me" etc etc.

I'm not sure what to tell you, except I fall squarely on the "old" side of that divide, having had to nurse Wordpress installations for nigh on 10 years now.

But every time -- every time -- an uncached Wordpress blog is linked to and dies with the famously unhelpful "Error establishing a database connection", somebody pops up to mention WP Supercache and/or W3 Total Cache.

Actually, if I have a pet peeve, it's that non-terrible caching isn't part of the Wordpress core. Probably breaks on gawdawfulhost.com or something, god forbid that 99.999% of the internet be better off from core architectural improvements when we could be working on the fifteenth new admin redesign!!1!

Edit: I realise now that you weren't talking about Wordpress and thus, my own pet obsession is clearly revealed.

Re: Making a site that can handle #1 on Hacker News

#24

Post by zalew » Mon, Apr 29, 2013, 4:13 AM UTC

kennywinker wrote: post #9

While the sentiment is clear, I had an uncached Wordpress site on shared hosting withstand #2 or #3 (I forget where exactly it peaked). HN isn't all that huge a traffic deliverer. It's just about the quality of that traffic.

> HN isn't all that huge a traffic deliverer. It's just about the quality of that traffic.

what?

Re: Making a site that can handle #1 on Hacker News

#25

Post by [deleted] » Mon, Apr 29, 2013, 4:14 AM UTC

saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

[deleted]

Re: Making a site that can handle #1 on Hacker News

#26

Post by tahoecoder » Mon, Apr 29, 2013, 4:15 AM UTC

jacques_chester wrote: post #23

tahoecoder wrote: post #22

Earlier quoted context omitted.

A lot of people on HN are still learning things like this. I understand that a post like this doesn't add any value to someone like you who probably knows this stuff inside and out, but I'm grateful for posts that give me insight into new tools that can help me. This isn't just a community for people with 10 years of programming experience. Should I not try to help out the community with blog posts about my experienc…

I feel like we're going to sail off into old/new user debate territory. "It's been done before" / "I'm new and so it's novel to me" etc etc. I'm not sure what to tell you, except I fall squarely on the "old" side of that divide, having had to nurse Wordpress installations for nigh on 10 years now. But every time -- every time -- an uncached Wordpress blog is linked to and dies with the famously unhelpful "Error estab…

Fair enough. I think we are all sick of hearing about WP Supercache.

My post was about middleman + s3 + cloudfront, however. I think this combination of tools isn't as well known and some people could benefit from knowing about them.

Re: Making a site that can handle #1 on Hacker News

#27

Post by jacques_chester » Mon, Apr 29, 2013, 4:15 AM UTC

zalew wrote: post #24

kennywinker wrote: post #9

While the sentiment is clear, I had an uncached Wordpress site on shared hosting withstand #2 or #3 (I forget where exactly it peaked). HN isn't all that huge a traffic deliverer. It's just about the quality of that traffic.

> HN isn't all that huge a traffic deliverer. It's just about the quality of that traffic. what?

He means that the readership of HN is high value. Name recognition amongst the HN readership might reasonably be considered to increase your chances of getting a nicer job, a book deal, a Y Combinator spot, a freelance contract, an investment round etc etc.

Re: Making a site that can handle #1 on Hacker News

#28

Post by jacques_chester » Mon, Apr 29, 2013, 4:19 AM UTC

tahoecoder wrote: post #26

jacques_chester wrote: post #23

Earlier quoted context omitted.

I feel like we're going to sail off into old/new user debate territory. "It's been done before" / "I'm new and so it's novel to me" etc etc. I'm not sure what to tell you, except I fall squarely on the "old" side of that divide, having had to nurse Wordpress installations for nigh on 10 years now. But every time -- every time -- an uncached Wordpress blog is linked to and dies with the famously unhelpful "Error estab…

Fair enough. I think we are all sick of hearing about WP Supercache. My post was about middleman + s3 + cloudfront, however. I think this combination of tools isn't as well known and some people could benefit from knowing about them.

In my opinion (given my superficial understanding of the prior situation), your problem was Apache + mod_php. The default settings for that combination are to chew memory until the bad people go away.

Out of curiousity, why middleman when you're already using Jekyll/Octopress?

(My dog in the static site generator fight is Nanoc, fwiw).

Re: Making a site that can handle #1 on Hacker News

#29

Post by xijuan » Mon, Apr 29, 2013, 4:25 AM UTC

kintamanimatt wrote: post #8

tahoecoder wrote: post #5

Earlier quoted context omitted.

I beg to differ. I changed a lot of stuff which was recommended by HN. What errors do you see? The main thing I didn't change was I kept the open source stuff highlighted because the vast majority of people visiting the site will want to find those resources.

"Get Perks at Local Restaurants & Bars" shouldn't be title cased. It should be "early beta" not "early Beta". The ellipsis after the "etc" should be a period. There are many others too. There were a ton of comments that were incredibly constructive and valid. I appreciate it's your site and you're not beholden to anybody, but almost all the criticism that was given was ignored. Also, you can edit comments on HN rathe…

I just saw the webpage for the first time. I agree with you. The writing is very poor on this webpage. Many sentences were awkwardly worded.

>Perks include offerings like free appetizers, complimentary drinks and more.

This is one of the awkward sentences that I found hard to comprehend at first..

And "early beta" is still not changed..

Re: Making a site that can handle #1 on Hacker News

#30

Post by tahoecoder » Mon, Apr 29, 2013, 4:26 AM UTC

jacques_chester wrote: post #28

tahoecoder wrote: post #26

Earlier quoted context omitted.

Fair enough. I think we are all sick of hearing about WP Supercache. My post was about middleman + s3 + cloudfront, however. I think this combination of tools isn't as well known and some people could benefit from knowing about them.

In my opinion (given my superficial understanding of the prior situation), your problem was Apache + mod_php. The default settings for that combination are to chew memory until the bad people go away. Out of curiousity, why middleman when you're already using Jekyll/Octopress? (My dog in the static site generator fight is Nanoc, fwiw).

My main site, AppRaptor, isn't on octopress. Just this blog. I agree with you 100% that the problem was apache + mod_php. Nginx would have probably been better. I decided to try out this s3 + cloudfront solution instead, though. I'll take a look at nanoc. Thanks for the info.