Earlier quoted context omitted.
I posted a bunch of test things and then deleted them.
I love this answer so much.
I really wanted to ask “How did you post things if the server was down?” but perhaps some things are better left as mysteries.
HN is up again
351–360 of 390 posts
Re: HN is up again
#352Earlier quoted context omitted.
I love this answer so much.
I really wanted to ask “How did you post things if the server was down?” but perhaps some things are better left as mysteries.
You could see them via HN’s API before they were deleted, nothing interesting; API was back up before the www.
Re: HN is up again
#353Earlier quoted context omitted.
Wow. It's possible that you have nailed this. Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible. But the two servers were set up at the same time , and it's…
This kind of thing is why I love Hacker News. Someone runs into a strange technical situation, and someone else happens to share their own obscure, related anecdote, which just happens to precisely solve the mystery. Really cool to see it benefit HN itself this time.
It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention:
HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments)
HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments)
Some HP Enterprise SSD will brick after 40000 hours without update - https://news.ycombinator.com/item?id=22697001 - March 2020 (1 comment)
HPE Warns of New Firmware Flaw That Bricks SSDs After 40k Hours of Use - https://news.ycombinator.com/item?id=22692611 - March 2020 (0 comments)
HPE Warns of New Bug That Kills SSD Drives After 40k Hours - https://news.ycombinator.com/item?id=22680420 - March 2020 (0 comments)
(there's also https://news.ycombinator.com/item?id=32035934, but that was submitted today)
Re: HN is up again
#354Re: HN is up again
#355Earlier quoted context omitted.
I really wanted to ask “How did you post things if the server was down?” but perhaps some things are better left as mysteries.
You could see them via HN’s API before they were deleted, nothing interesting; API was back up before the www.
Good observation. Posting something and then seeing it show up in the API was one of the things we were testing. It exercises a lot of the code.
Re: HN is up again
#356Earlier quoted context omitted.
I once had a small fleet of SSDs fail because they had some uptime counters that overflowed after 4.5 years, and that somehow persistently wrecked some internal data structures. It turned them into little, unrecoverable bricks. It was not awesome seeing a bunch of servers go dark in just about the order we had originally powered them on. Not a fun day at all.
You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.
It's concerning that a hosting company was unaware of the 40,000 hour situation with SSD it was deploying. Anyone in hosting would have been made aware of this, or at least should have kept a better grip on happenings in the market.
Re: HN is up again
#357Earlier quoted context omitted.
Oh my goodness yes. I had the "great" idea to use Azure Functions to do a task at work. It's **ing insane how difficult it is to specify an Azure Function all in code with reasonable CI/CD, AD permissions, logging, and dev/prod instances. I wrote about what it takes at https://www.bbkane.com/blog/azure-functions-with-terraform/ but the experience really soured me on cloud services.
Is this... a deliberate attempt at constructing a Rube Goldberg machine? In all seriousness, at least 2/3rds of the complexity is because of your choice of tools and approach. Terraform alone makes things significantly more complex. If you just want to trigger a deployment, then a Template Spec made from a Bicep file could be banged out in like... an hour.[1] When in Rome, do as the Romans do. You basically took a Mi…
> Is this... a deliberate attempt at constructing a Rube Goldberg machine?
> I normally bill for cloud automation advice, but the gist is
Can you please omit supercilious swipes from your comments here? Everybody knows different things. If you know more than someone else about $thing, that's great—but please don't put them down for it. That's not in the spirit of kindness and curious conversation that we're hoping for here.
https://news.ycombinator.com/newsguidelines.html
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
Re: HN is up again
#358Earlier quoted context omitted.
So that means dataloss.. Probably restored from backup. Good news for people who were banned, or for posts that didn't get enough momentum :) edit: Was restored from backup.. so def. dataloss
8 hours of downtime, but not data loss, since there was no data to lose during the downtime. Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565 First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?) So, 7h 45m of downtime. What we don't know is how many posts (or vo…
[deleted]
Re: HN is up again
#359Earlier quoted context omitted.
Curiosity got the better of me. Why was there a 6 ID gap between the last post and first post? The answer seems to be that admins were making posts, which is neat. (There was also one lonely Flexport job ad.) Is your backup system tied to your API? Algolia is a third party service, and streaming the latest HN data to Algolia seems pretty similar to streaming it to a backup system.
i got that Flexport ad too.. haha kinda alarming if they are the only YC company still hiring
Btw, job ads get queued long in advance and then the software picks the next one when it's time for a job ad. After 8 hours of being down, the software thought it was time for a job ad.
Re: HN is up again
#360Earlier quoted context omitted.
Oh that is interesting, I guess they just spun up a beefy EC2 instance. I'm noticing slower performance, I used to get about NetRange: 50.112.0.0 - 50.112.255.255 CIDR: 50.112.0.0/16 NetName: AMAZON-EC2-USWESTOR NetHandle: NET-50-112-0-0-1 Parent: NET50 (NET-50-0-0-0-0) NetType: Direct Allocation OriginAS: AS14618 Organization: Amazon.com, Inc. (AMAZO-47)
I hope it's temporary. Would hate HN to move to the "cloud" from bare metal.
Us too.