Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Surviving a traffic surge: Three techniques to scale your site fast

matt.might.net

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Surviving a traffic surge: Three techniques to scale your site fast

#21

Post by JshWright »

coderdude wrote: post #7

>>(Step 1: Cut image quality) Page load time dropped from 24 to 12 seconds. Wow. I had no idea that could make such a difference. I suppose the issue was with the low number of threads set in the Apache configuration. The server was spending its time sending out static content when it could have been doing more important things? I signed up with S3 to serve up my static content to keep that load off my server. I shou…

Linode offers StackScripts, which can automate a lot of the initial deployment steps. http://www.linode.com/stackscripts/

Re: Surviving a traffic surge: Three techniques to scale your site fast

#22

Post by edu »

yuvadam wrote: post #10

I'm not at all familiar with Linode's iPhone app - but the bottom graph looks an awful lot like a system load graph, and not a CPU utilization graph. "CPU utilization never exceeded 3%"? Really? Maybe the system load was at 3.0 for a few days?

Completely agree, that a system load graph not CPU usage.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#23

Post by jedsmith »

yuvadam wrote: post #10

I'm not at all familiar with Linode's iPhone app - but the bottom graph looks an awful lot like a system load graph, and not a CPU utilization graph. "CPU utilization never exceeded 3%"? Really? Maybe the system load was at 3.0 for a few days?

It is CPU utilization. Linodes have access to four cores so the graph's scale is from 0% to 400%, which might not necessarily synchronize with top. Same information as the Linode Manager, just a different time scale to fit constraints of the iPhone screen. The same graph on my personal Linode, from the Linode Manager, has a legend that identifies itself as percentage:

http://dropbox.jedsmith.org/undertow_hn.png

That number comes from outside of the Linode, not within it. The 14-day graph is also a 2-hour average, which means each value is over a particularly large period and big spikes will be smoothed out; I don't think the results are too far from the real world here, though. Not a lot of CPU time is required to fetch things from various places and transmit over the network...it's mostly wait time.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#24

Post by jasonkester »

sp332 wrote: post #16
jasonkester wrote: post #12

No. Stop it. Never ever scale your blog. It sounds like the author was making the same mistake that pretty much everybody makes: Treating your blog as though it were dynamic content. But it's not. It's static HTML, and you should never have to make any modifications to anything to make it scale. Step one: Have your blog export all entries to plain HTML. Step two (optional): move your imagery out to S3/Cloudfront. Tha…

If the blog supports comments, it needs to talk to databases. Step one: Have your blog export all entries to plain HTML. He did that. A website hosting a blog, on the other hand, needs to serve files. And that's been a solved problem for fifteen years. "Too few Apache threads" is a known problem, which he recognized as soon as he saw the load numbers.

He did that.

Exactly my point. He did that and it worked.

If you have a blog that may one day see traffic, I'd recommend taking that bit of the post to heart and skipping the whole "serve blog content via the database" part altogether.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#25

Post by jedsmith »

JoachimSchipper wrote: post #4
bmelton wrote: post #2

Those tips are obviously worthwhile, but doesn't address massive scale for heavily dynamic sites. These should definitely be at the beginning of any optimization checklist for sure. One thing I found interesting was his remark this: Google Analytics failed to detect the surge: page load time was so high that visitors were closing the page before analytics could load. He then remarks that it took Analytics 15 hours to…

These tips are for people getting slashdotted, not people building the next Google. Your average blog simply isn't "heavily dynamic", and if you're building Google/Facebook/... there are better resources. Google Analytics is Javascript-based and thus vulnerable to people closing the page before it loads. It's actually near-realtime, but just doesn't work if you're having this problem. I'd imagine watching free Apache…

> Google Analytics is Javascript-based and thus vulnerable to people closing the page before it loads.

And people with Javascript disabled, which is a startling number. The disparity between reported views in Analytics and what I can observe from my Web server logs is amazingly large.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#26

Post by pdenya »

bmelton wrote: post #2

Those tips are obviously worthwhile, but doesn't address massive scale for heavily dynamic sites. These should definitely be at the beginning of any optimization checklist for sure. One thing I found interesting was his remark this: Google Analytics failed to detect the surge: page load time was so high that visitors were closing the page before analytics could load. He then remarks that it took Analytics 15 hours to…

Google Analytics updates hourly, reporting gives you only full days by default but you can see data much more up to date than that.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#28

Post by JoachimSchipper »

jedsmith wrote: post #25
JoachimSchipper wrote: post #4

Earlier quoted context omitted.

These tips are for people getting slashdotted, not people building the next Google. Your average blog simply isn't "heavily dynamic", and if you're building Google/Facebook/... there are better resources. Google Analytics is Javascript-based and thus vulnerable to people closing the page before it loads. It's actually near-realtime, but just doesn't work if you're having this problem. I'd imagine watching free Apache…

> Google Analytics is Javascript-based and thus vulnerable to people closing the page before it loads. And people with Javascript disabled, which is a startling number. The disparity between reported views in Analytics and what I can observe from my Web server logs is amazingly large.

Are you sure those aren't bots? Analytics packages vary widely in their ability to filter out bots.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#29

Post by commx »

edu wrote: post #22
yuvadam wrote: post #10

I'm not at all familiar with Linode's iPhone app - but the bottom graph looks an awful lot like a system load graph, and not a CPU utilization graph. "CPU utilization never exceeded 3%"? Really? Maybe the system load was at 3.0 for a few days?

Completely agree, that a system load graph not CPU usage.

No, it's CPU usage, not a load average.

Re: Surviving a traffic surge: Three techniques to scale your site fast

#30

Post by jedsmith »

JoachimSchipper wrote: post #28
jedsmith wrote: post #25

Earlier quoted context omitted.

> Google Analytics is Javascript-based and thus vulnerable to people closing the page before it loads. And people with Javascript disabled, which is a startling number. The disparity between reported views in Analytics and what I can observe from my Web server logs is amazingly large.

Are you sure those aren't bots? Analytics packages vary widely in their ability to filter out bots.

Bots have Javascript disabled too. :)

You're right, of course. I considered it and started grepping them out, but got bored and did something else interesting. From a casual glance, there were a lot more Mozilla UAs than reported views, though.

Post reply on HN