Tell HN: Server Status

#1

Post by kogir » Thu, Jan 16, 2014, 12:37 PM UTC

HN went down for nearly all of Monday the 6th. I suspected failing hardware.

I configured a new machine that is nearly identical to the old one, but using ZFS instead of UFS. This machine can tolerate the loss of up to two disks. I switched over to it early morning on the 16th, around 1AM PST.

Performance wasn't great. Timeouts were pretty frequent. I looked into it quickly, couldn't see anything obvious, and decided to sleep on it. I switched back to the old server, expecting to call it a night.

Then the old server went down. Again. The filesystem was corrupted. Again. So I switched back to the new server. During this switch some data was lost, but hopefully no more than an hour.

And here we are. I'm sorry that performance is poor, but we're up. I'll work to speed things up as soon as I can, and I'll provide a better write-up once things are over. I'm also really sorry for the data loss, both on the 6th and today.

Re: Tell HN: Server Status

#2

Post by nmc » Thu, Jan 16, 2014, 12:42 PM UTC

Thanks for the info!

Out of curiosity, do you have an idea about the source of the corruption problems?

Re: Tell HN: Server Status

#3

Post by jffry » Thu, Jan 16, 2014, 1:02 PM UTC

Thanks for the writeup.

Re: Tell HN: Server Status

#4

Post by hartator » Thu, Jan 16, 2014, 1:02 PM UTC

Not really related but any update on releasing the HN code again?

[the current release is pretty old: https://github.com/wting/hackernews]

Re: Tell HN: Server Status

#5

Post by hartator » Thu, Jan 16, 2014, 1:03 PM UTC

nmc wrote: post #2

Thanks for the info! Out of curiosity, do you have an idea about the source of the corruption problems?

The OP is saying "the loss of up to two disks", maybe a hard drive failure?

Re: Tell HN: Server Status

#6

Post by cincinnatus » Thu, Jan 16, 2014, 1:06 PM UTC

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

Re: Tell HN: Server Status

#7

Post by ishener » Thu, Jan 16, 2014, 1:06 PM UTC

may i ask where are the machines hosted? is that on AWS? if not, why don't you move to a more reliable hosting, like AWS?

Re: Tell HN: Server Status

#8

Post by ishener » Thu, Jan 16, 2014, 1:08 PM UTC

ishener wrote: post #7

may i ask where are the machines hosted? is that on AWS? if not, why don't you move to a more reliable hosting, like AWS?

from the downvotes i gather i may not ask...

Re: Tell HN: Server Status

#9

Post by raimue » Thu, Jan 16, 2014, 1:09 PM UTC

cincinnatus wrote: post #6

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

It's 2014 already!

Re: Tell HN: Server Status

#10

Post by oneeyedpigeon » Thu, Jan 16, 2014, 1:10 PM UTC

cincinnatus wrote: post #6

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

HN isn't a mission-critical business. It's a fun, experimental distraction (albeit one that we all rate very highly, of course).