Earlier quoted context omitted.
Crazy that Dang literally manages HN in his sleep! We all knew that but I haven't seen any confirmation before this.
failing to manage HN in my sleep is more like it
And that is a good thing. Sleep tight!
Tell HN: HN was down
181–190 of 343 posts
Re: Tell HN: HN was down
#182Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
Even after providing firebase endpoint, crawlers come to the site ?
Re: Tell HN: HN was down
#183I got stuck in an infinite loop. Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
Sometimes I'll catch myself absentmindedly reopening the browser and checking two or three front pages, seconds after having just checked them and closed the browser.
That's a sign of addiction and I highly recommend changing your behaviour towards those pages!
Re: Tell HN: HN was down
#184Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
I didn't realize you were carrying the pager too! Kudos!
I feel such a sense of kinship for anyone who carries a pager, almost 7 years at my current role doing it. Super cool that dang is among our number :)
Re: Tell HN: HN was down
#185Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users
How does this happen?
Re: Tell HN: HN was down
#186Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
Can't speak for others, but I'm sure i'll be pretty fine if no one gets woken up if HN is down...
Of course, they'd better restore service after they wake up naturally, because I need my HN dose. But it's not worth losing sleep over it.
Re: Tell HN: HN was down
#187TIL I have a "open Hacker News" hand reflex
What? You mean you ... close the HN tab?
Re: Tell HN: HN was down
#188Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
Maybe it would be fine if ops alerts were silenced during normal US sleeping hours?
HN is important, but unlikely much harm could be done before morning.
(Source: Lost a lot of sleep at one place, enough to realize that sleep interruption and deficit has significant costs.)
Re: Tell HN: HN was down
#189Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users How does this happen?
How does this happen?
Not the person you are asking. Bot operators have an incentive to make crawlers look as much like a human as possible so they do not get blocked. Some of them fail miserably and some nearly succeed. That makes it trivial to accidentally block a real person. I am personally fine with that given I do not pay for this site and have no SLA or contract with it.
Re: Tell HN: HN was down
#190Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!