Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashdot effect

en.wikipedia.org

Post reply on HN
Print view View original

21–30 of 74 posts

Re: Slashdot effect

#21

Post by Sesse__ »

JoshTriplett wrote: post #9
lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

Hit a webserver with many thousands or tens of thousands of requests per second, and whatever the weakest link is will break: memory usage, CPU usage, database, bandwidth, billing, third-party service quotas...

At the time when the Slashdot effect was at its peak, the limiting factors were probably RAM or bandwidth (third-party services was not as much a thing, and CPUs were generally able to push 10 Mbit/sec without much of a problem -- but 10 Mbit/sec Internet lines would be a luxury). Like, you could reasonably host Apache on 16 MB of RAM on a 2 Mbit/sec line for your hobby site with a couple of photos of your car or whatever, and then ten thousand people would come along at once and find it interesting, and your site would be dead for a while.

Re: Slashdot effect

#22

Post by mbreese »

metajack wrote: post #10
lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

Many blogs are completely dynamic and grab content out of a database on every request. The static site generation style fell out of early fashion when wordpress took over from moveable type, and didn't really return much until jekyll. Even today I think most blogs you see are mostly dynamic with the platforms using caching to avoid hitting the database every time. Most people don't do performance tuning of their blog…

Don’t forget - at the time, even if you were pulling a static file, you were pulling it from disk. And disk was much slower… IDE was still common. If you had a good server, it could have been SCSI, but a white label Linux box could be quite slow. So, unless you had a good caching setup, you could still have issues sending even static data.

Also, when slashdotting was common, networks weren’t always the most robust. It wasn’t uncommon in the late 90’s for companies to have a single a T1 line for access to the internet. So, unless you had a good, well peered network provider, that was another potential bottleneck.

We worked with what we had. Thankfully, everything is more robust now.

Re: Slashdot effect

#23

Post by romanhn »

theideaofcoffee wrote: post #16
SirFatty wrote: post #4

Earlier quoted context omitted.

yeah, it's around, but a ghost of its former glory.

It's sad to see its decline. I started reading right around 2000 and even noticed that it started to decline even as early as 2008, after and around some of its redesigns. I know I quit regularly stopping around there probably around that time too.

Yup, around 2008 is when I noticed that many of the Slashdot front page submissions were discussed on this new site called Hacker News 1-2 days prior, minus the trolling and the comedy threads. It was a pretty easy switch at that point.

Re: Slashdot effect

#24

Post by [deleted] »

aleph_minus_one wrote: post #8
lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

> Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. At that time, it was quite common to start a new thread or even fork a new process per request/connection. Apache was prone to this: > https://www.digitalocean.com/commu…

[deleted]

Re: Slashdot effect

#25

Post by 0xbadcafebee »

lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

That's what's possible today. It wasn't back then (see: https://en.wikipedia.org/wiki/C10k_problem)

The other reason is, not every web server, web application, database, etc is optimized for large number of connections. Sometimes they take up too much CPU and memory. Sometimes each page request triggers a connection to a database, so the database connection limit is hit. Sometimes the queries are inefficient and slow the database to a crawl. Sometimes the sheer number of packets causes so many interrupts that it sucks CPU away from the application/database. And sometimes the network connection of the webserver was just slow.

Most people hit by the /. effect were not just a simple web server serving static content. Often it was a php website with a mysql database, on a server less powerful than a 2015 smartphone.

Re: Slashdot effect

#27

Post by locknitpicker »

lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

> Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware.

You're grossly overstating the capacity of a server to meet peak load demands, specially when hit with unexpected load spikes. The slashdot effect was a phenomenon observed two decades ago, in a time when the bulk of the web was served as Apache running CGI scripts. The rate of requests any random server could process was around ~300requests per second, and some of the links being hugged to death were literally served from a box under some dude's desk.

Re: Slashdot effect

#28

Post by joshdavham »

virgil_disgr4ce wrote: post #3

Kind of funny to find this on the front page of HN. Makes me wonder what percentage of today's HN readers didn't live through the Slashdot era. (I'm aware it's still around.)

> Makes me wonder what percentage of today's HN readers didn't live through the Slashdot era.

I definitely missed out :(

What was it like?

Re: Slashdot effect

#30

Post by dakna »

lionkor wrote: post #2

Can someone with some more experience explain why this happens? The most basic webserver, like a TCP socket served from a C program, serving some files etc. can take tens of thousands of requests per second, per core, on reasonably recent hardware. What's going on that's taking down these sites, really, when its not a quota limit?

At the time, a standard LAMP stack setup wasn't prepared for the https://en.wikipedia.org/wiki/C10k_problem. Quite often you had no DB connection pool and every request for dynamic data opened/closed a DB connection. Having a fixed number of connections available also quickly hit the limits unless it was a dynamically configured pool. Which also filled up eventually.

If you were lucky you could reduce load by caching the URLs taking up the most resources and generate a webpage copy in the filesystem, then add some URL rewrite logic to Apache to skip going through your application logic and bypass the DB.

Then you discovered there is a limit of open file descriptors in Linux. After updating this and also shutting down all log files you ran out of things to change pretty quick and started pricing a beefier server. For next time.

Post reply on HN