Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making a site that can handle #1 on Hacker News

blog.appraptor.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Making a site that can handle #1 on Hacker News

#31

Post by jacques_chester »

tahoecoder wrote: post #30
jacques_chester wrote: post #28

Earlier quoted context omitted.

In my opinion (given my superficial understanding of the prior situation), your problem was Apache + mod_php. The default settings for that combination are to chew memory until the bad people go away. Out of curiousity, why middleman when you're already using Jekyll/Octopress? (My dog in the static site generator fight is Nanoc, fwiw).

My main site, AppRaptor, isn't on octopress. Just this blog. I agree with you 100% that the problem was apache + mod_php. Nginx would have probably been better. I decided to try out this s3 + cloudfront solution instead, though. I'll take a look at nanoc. Thanks for the info.

Nanoc is hard to get used to but as far as I'm concerned it's a bucket filled with magic.

Re: Making a site that can handle #1 on Hacker News

#32

Post by tahoecoder »

saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

Not sure why I have to defend the number. I think any #1 post would see 287 concurrents. http://imgur.com/yfn9Dnd (look at the 30 day max). I was a little off, it was 282 concurrents.

Re: Making a site that can handle #1 on Hacker News

#33

Post by bradhe »

saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

> "...so I just put the site on my Linode 512 (512 Mb of RAM). I usually do most of my development in Ruby and Rails and deploy to Heroku but I was lazy and already had a linode setup and configured for PHP."

That will get real sideways real fast. I do concur that 287 concurrent requests with 3k page views over a day doesn't even make sense mathematically.

Re: Making a site that can handle #1 on Hacker News

#34

Post by saurik »

tahoecoder wrote: post #32
saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

Not sure why I have to defend the number. I think any #1 post would see 287 concurrents. http://imgur.com/yfn9Dnd (look at the 30 day max). I was a little off, it was 282 concurrents.

It seems that there was good reason to challenge your numbers: you are using "concurrent users hitting my site" in a way that people in a website optimization context would not ever expect. This screenshot is talking about "concurrent visits", and has a figure for the average time that users were "engaged" (over 45 minutes). These are not performance measurements: they are the kinds of metrics you use to determine how interesting and sticky the content of your site is (along with things like your bounce rate and content flow).

In fact, this number (apparently a feature of Chartbeat) claims to be measuring "concurrent people sitting at a computer looking at pages from your site", not concurrent requests or "hits" to your webserver, or even concurrent HTTP connections (which may be idle for long periods of time). This number is almost entirely meaningless for the purposes of discussing your site's load. Imagine an HTML5 JavaScript game that took an entire day to play: with one request per second you may find yourself with tens of thousands of "concurrent visits".

Re: Making a site that can handle #1 on Hacker News

#35

Post by tahoecoder »

saurik wrote: post #34
tahoecoder wrote: post #32

Earlier quoted context omitted.

Not sure why I have to defend the number. I think any #1 post would see 287 concurrents. http://imgur.com/yfn9Dnd (look at the 30 day max). I was a little off, it was 282 concurrents.

It seems that there was good reason to challenge your numbers: you are using "concurrent users hitting my site" in a way that people in a website optimization context would not ever expect. This screenshot is talking about "concurrent visits", and has a figure for the average time that users were "engaged" (over 45 minutes). These are not performance measurements: they are the kinds of metrics you use to determine ho…

Knowing how many people are on your pages at any one time isn't a meaningless number. You're right in the sense that it's not a great way to assess quantitative site load. I just used that number in the blog to give a qualitative assessment of how many people were currently on my site.

Also, you are misunderstanding this chartbeat number. These 282 visitors weren't spread out over 45 minutes. These are real time users. Granted, some users are idle but most are interacting with the pages.

If this blog post were about the specifics of server load from a #1 HN post, I wouldn't even use the chartbeat figure. The post was about middleman + s3 + cloudfront. The chartbeat figure was more than enough to give readers a qualitative glance of how many people were looking at my site.

Re: Making a site that can handle #1 on Hacker News

#37

Post by paulsutter »

saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

Surely there's an opportunity here. It's mind boggling how people are having performance issues here, but they are. It's an opportunity for someone to make some money and improve the internet, by fixing whatever tools they're using.

To amplify your comment, processors today process billions of instructions per second. Even if all 3000 pageviews _did_ hit within one minute, thats hundreds of millions of instructions available per pageview. His pages just aren't complex enough to require that many instructions to serve.

tahoecoder's image to "prove" his load indicates he had 287 visits within a 45 minute window. Allowing hundreds of _billions_ of instructions per page served. Please do give me a break.

At Quantcast we handle 800,000 HTTP requests per second, and process 30 petabytes a day, so it really is possible to handle actual high loads.

Re: Making a site that can handle #1 on Hacker News

#38

Post by saurik »

tahoecoder wrote: post #35
saurik wrote: post #34

Earlier quoted context omitted.

It seems that there was good reason to challenge your numbers: you are using "concurrent users hitting my site" in a way that people in a website optimization context would not ever expect. This screenshot is talking about "concurrent visits", and has a figure for the average time that users were "engaged" (over 45 minutes). These are not performance measurements: they are the kinds of metrics you use to determine ho…

Knowing how many people are on your pages at any one time isn't a meaningless number. You're right in the sense that it's not a great way to assess quantitative site load. I just used that number in the blog to give a qualitative assessment of how many people were currently on my site. Also, you are misunderstanding this chartbeat number. These 282 visitors weren't spread out over 45 minutes. These are real time user…

You had ~2250 visits that day (1440 minutes), each of which spent (on average) ~45 minutes looking at your website. Assuming only one concurrent visitor, you'd have had 1440/45 = 32 total visits. You actually had 2250/32 = ~70 times that number, so you averaged ~70 concurrent visits that day.

Due to expected fluctuations (day/night, when you posted to HN) there will be considerable real-time variation; it is easy to believe that you spiked up to 282 concurrent visitors at some point when people were heavily commenting on HN: however, the real question here is what the concurrent number of requests looked like.

Finally, yes: one way to look at these visitors is that they were "spread out over 45 minutes" (although that isn't how I'd describe it myself). If you asked "during any given 45 minute period, how many visits (on average) started during that period", we would be looking at 2250/(1440/45), or ~70 visits starting in that window.

Re: Making a site that can handle #1 on Hacker News

#39

Post by tahoecoder »

paulsutter wrote: post #37
saurik wrote: post #21

Your site apparently got ~2250 visits per day (so, less than two per minute) at the height of your "surge", and seems to consist of three pages (/, /about, and /open-source). Most people are only going to look at just /, so let's say 3000 pageviews. The day after was still seeing good amounts of traffic, so it wasn't some kind of momentary "all 3000 hit the site in the same minute" situation: it seems like a fairly b…

Surely there's an opportunity here. It's mind boggling how people are having performance issues here, but they are. It's an opportunity for someone to make some money and improve the internet, by fixing whatever tools they're using. To amplify your comment, processors today process billions of instructions per second. Even if all 3000 pageviews _did_ hit within one minute, thats hundreds of millions of instructions a…

Lots of people are misunderstanding that chartbeat figure. It's not 287 visits in a 45 minute window. These are visitors who are currently interacting with your site. Some are idle but most aren't.

Re: Making a site that can handle #1 on Hacker News

#40

Post by tahoecoder »

saurik wrote: post #38
tahoecoder wrote: post #35

Earlier quoted context omitted.

Knowing how many people are on your pages at any one time isn't a meaningless number. You're right in the sense that it's not a great way to assess quantitative site load. I just used that number in the blog to give a qualitative assessment of how many people were currently on my site. Also, you are misunderstanding this chartbeat number. These 282 visitors weren't spread out over 45 minutes. These are real time user…

You had ~2250 visits that day (1440 minutes), each of which spent (on average) ~45 minutes looking at your website. Assuming only one concurrent visitor, you'd have had 1440/45 = 32 total visits. You actually had 2250/32 = ~70 times that number, so you averaged ~70 concurrent visits that day. Due to expected fluctuations (day/night, when you posted to HN) there will be considerable real-time variation; it is easy to…

Creating a daily average of concurrent visits isn't relevant. Servers don't take an average of all the possible future hours of complete inactivity when they are serving up requests.

The post was only up at the top for a couple of hours before the server performance got so bad such that everyone started flagging it. My post then dropped off the front page and quickly went to page 4 or 5 due to the flagging.

I agree with you that the more interesting number would be the the concurrent number of requests.

Post reply on HN