It would be a nice feature if hackernews had a simple news.ycombinator.com/status page that I could trust. I don't like loading those some-sketchy-domain/status-right-now sites.
Tell HN: HN was down
191–200 of 343 posts
Re: Tell HN: HN was down
#192I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.
[flagged]
What a rude thing to post. Hund, don't listen to this entitled nonsense. There is a reason it's called human error. Companies 100x your size and 10000x your revenue like AWS, Microsoft, CloudFlare, CrowdStrike can't figure out how to accurately provide status dashboards. At least you took the time to explain your mistakes. If anything you got another supporter for your honesty.
Re: Tell HN: HN was down
#193Earlier quoted context omitted.
I'm glad I'm not the only one. If I type 'n' into any browser it autocompletes to HN.
Save typing hundreds of letters per day, and replace about:newtab with news.ycombinator.com, now you can just do CTRL+T :)
at that moment my productivity would drop to zero
Re: Tell HN: HN was down
#194Earlier quoted context omitted.
I didn't realize you were carrying the pager too! Kudos!
I feel such a sense of kinship for anyone who carries a pager, almost 7 years at my current role doing it. Super cool that dang is among our number :)
Yep, have been on constant "pager duty" for 2+ years, although I have more help now and I get paged 1-3 times a week instead of per night. Still, carry my lappy everywhere I go. Bought an ARM Windows laptop to get that 20hr battery life so I could worry less during my travels. You know, fancy things like going get food or going grocery shopping.
Re: Tell HN: HN was down
#195Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
Even after providing firebase endpoint, crawlers come to the site ?
Unfortunately, the firebase API is very bad as they even acknowledge that in their github page.
Re: Tell HN: HN was down
#196Earlier quoted context omitted.
failing to manage HN in my sleep is more like it
We all have our moments, and I personally consider HN to be “best effort”, almost like a volunteer project. I’m not certain I’m correct: but thats the optics I have so my expectations are adjusted to that. So don’t beat yourself up please. When I worked for “SaaS unicorn” we typically had multiple levels of escalation, and acknowledging would have done nothing because the alarm would continue firing until fixed. Not…
This. If it were a business-critical money fountain, I'd expect follow-the-sun SRE coverage. I don't think it is, so I can probably accept drinking my morning coffee without scrolling HN once in a while. There's only so much one can beat oneself up about a slow/incorrect response when the on-call is handled by what, just one person? maybe two people in the same time zone?
(Might be wise though to have PagerDuty configured to re-alert if the outage persists.)
Re: Tell HN: HN was down
#197I got stuck in an infinite loop. Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
HN is how I discover whether other sites are down or not, so it serves a critical function, so of course I check it frequently.
/s
Re: Tell HN: HN was down
#198Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
Even after providing firebase endpoint, crawlers come to the site ?
Most crawlers have no concept of what that is. They will follow links to this site and then follow links out of this site even after being told not to [1]. The majority of crawlers follow zero rules, RFC's, etc... The few platforms that do follow standards and rules are akin to a law abiding citizen in Mos Eisley.
[1] - rel="nofollow"
Re: Tell HN: HN was down
#199Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
During the last week my IP was banned for unknown reason. Glad to hear it might not be a problem from my side.
Re: Tell HN: HN was down
#200Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago. (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.) In this case, we also had a secondary failure: PagerDuty woke me up at 5:24…
I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!
I can’t put my finger on anything within the last decade, but I seem to recall it running in something close to its current form on a single core on a single server for a long time:
https://news.ycombinator.com/item?id=5229522
Re: traffic, dang said (2022):
https://news.ycombinator.com/item?id=33454140
I took it as a good reminder that the hard part is the human part: that high-overhead features and UI fripperies are nice but not necessary (or sufficient) to keep a community healthy and vibrant over the decades.
(And on the subject of the human side, if you didn’t catch Anna Wiener’s 2019 profile, it’s here:
https://www.newyorker.com/news/letter-from-silicon-valley/th... )