What caused that outage

#1

Post by pg » Thu, Mar 12, 2009, 1:20 AM UTC

HN was down today for around 2 hours. Sorry about that.

The News server currently crashes a couple times a day when it runs out of memory. All the comments and stories no longer fit in the 2 GB we can get on a 32 bit machine. We'd been planning to upgrade to a new 64 bit server. In the meantime it was arguably a rather slow form of GC.

Unfortunately the process somehow got wedged in the middle of segfaulting. We're not sure why and will probably never know. But that meant the process that usually notices when News is wedged and restarts it was unable to kill it.

Fortunately rebooting the machine solved the problem. But now we'll presumably be switching to that new, bigger server sooner rather than later.

As far as I can tell it was a coincidence that this happened today. It doesn't seem to have been caused either by the increased traffic, or the excessive number of posts about Erlang.

Re: What caused that outage

#2

Post by [deleted] » Thu, Mar 12, 2009, 1:24 AM UTC

[deleted]

Re: What caused that outage

#3

Post by mixmax » Thu, Mar 12, 2009, 1:45 AM UTC

That's what happens when you make something in a new and unproven language.

And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language.

It's great to see people eat their own dogfood.

Re: What caused that outage

#4

Post by dylanz » Thu, Mar 12, 2009, 1:50 AM UTC

mixmax wrote: post #3

That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.

Wow, had no idea this was written in Arc. +1 to what mixmax said. It's the only way to keep pushing the envelope.

Re: What caused that outage

#5

Post by acegopher » Thu, Mar 12, 2009, 1:55 AM UTC

The server goes down, and when it comes back up the front page is filled with stories about Erlang. Coincidence?

Re: What caused that outage

#6

Post by jm4 » Thu, Mar 12, 2009, 1:57 AM UTC

Bummer. Maybe you should consider rewriting in Erlang... :)

Re: What caused that outage

#7

Post by mixmax » Thu, Mar 12, 2009, 2:01 AM UTC

dylanz wrote: post #4

mixmax wrote: post #3

That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.

Wow, had no idea this was written in Arc. +1 to what mixmax said. It's the only way to keep pushing the envelope.

Boldness drives innovation

Re: What caused that outage

#8

Post by mblakele » Thu, Mar 12, 2009, 2:02 AM UTC

Why was the site using a 32-bit environment to begin with? Opterons have been cheap for years now, and all new Xeons are capable of 64-bit operation.

Re: What caused that outage

#9

Post by natrius2 » Thu, Mar 12, 2009, 2:07 AM UTC

Speaking of bugs, pages don't load for me when I'm logged into my real account (without the 2). The front page cuts off after "1." and comments pages cut off right after the first sub-table opens. The leaders and submit pages load fine.

I thought it was just a sign that I should get back to work, but maybe my account got put into some sort of inconsistent state?

Re: What caused that outage

#10

Post by MrRage » Thu, Mar 12, 2009, 2:13 AM UTC

mblakele wrote: post #8

Why was the site using a 32-bit environment to begin with? Opterons have been cheap for years now, and all new Xeons are capable of 64-bit operation.

What chip in put out in the last 2 years or so couldn't run a 64 bit system? I'm pretty sure most of the AMD chips could... Edit: By chip I mean one for a desktop CPU, e.g. Atom don't count.