Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HN is up again

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 390 posts

Re: HN is up again

#291

Post by toast0 »

GekkePrutser wrote: post #222
toast0 wrote: post #190

Earlier quoted context omitted.

I also don't know about literature on this phenomenon, but i recall HP had two different SSD recalls because when the uptime counter rolled over, they would fail. That's not even load dependent, just did you get a batch and power them on all at the same time. Uptime is too high causing issues isn't that unusual for storage, unfortunately. It's not always easy, but if you can, you want manufacturer diversity, batch di…

The crucial M4 had this too but it was fixable with a firmware update. https://www.neoseeker.com/news/18098-64gb-crucial-m4s-crashi...

That one looks not too bad, seems like you can fix it with a firmware update after it fails. A lot of disk failures due to firmware bugs end up with the disk not responding to the bus, so it becomes somewhere between impossible and impractical to update the firmware.

Re: HN is up again

#293

Post by jiggawatts »

bbkane wrote: post #243
geoffeg wrote: post #173

Earlier quoted context omitted.

I've wondered about this a lot recently as I've spent a lot of time recently fighting with Terraform, Atlantis, CircleCI, github and AWS. The first half of my career was deploying code to various UNIX machines. When there was a problem I could login to that machine and use various (common) tools to diagnose the issue. I may not have had root but I was at least able to gain insight into what was likely the culprit. Th…

Oh my goodness yes. I had the "great" idea to use Azure Functions to do a task at work. It's **ing insane how difficult it is to specify an Azure Function all in code with reasonable CI/CD, AD permissions, logging, and dev/prod instances. I wrote about what it takes at https://www.bbkane.com/blog/azure-functions-with-terraform/ but the experience really soured me on cloud services.

Is this... a deliberate attempt at constructing a Rube Goldberg machine?

In all seriousness, at least 2/3rds of the complexity is because of your choice of tools and approach. Terraform alone makes things significantly more complex. If you just want to trigger a deployment, then a Template Spec made from a Bicep file could be banged out in like... an hour.[1]

When in Rome, do as the Romans do. You basically took a Microsoft product and tried to automate it with a bunch of Linux-native tools. Why would you think this would be smooth and efficient?

Have you ever tried automating Linux with VB Script? This is almost the same thing.

[1] Someone had a similar example here using Logic Apps and a Template Spec: https://cloudjourney.medium.com/azure-template-spec-and-logi...

Re: HN is up again

#294

Post by dang »

pastor_bob wrote: post #48
sillysaurusx wrote: post #20

HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 Double disk failure is improbable but not impossible. The most impressive thing is that there seems to be no dataloss, almost whatsoever. Whatever the backup system is, it seems rock solid.

is dang pushing changes and such on his own? sounds like it is run by one guy

I push changes on my own all the time, but the work of getting HN running again today was overwhelmingly done by my brilliant colleague mthurman.

Re: HN is up again

#295

Post by dang »

jbverschoor wrote: post #79
Cerium wrote: post #46

Earlier quoted context omitted.

I came to the same conclusion by observing that there are posts and comments from only eight hours ago.

So that means dataloss.. Probably restored from backup. Good news for people who were banned, or for posts that didn't get enough momentum :) edit: Was restored from backup.. so def. dataloss

8 hours of downtime, but not data loss, since there was no data to lose during the downtime.

Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565

First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?)

So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post.

There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds.

I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.

Re: HN is up again

#296

Post by bbkane »

jiggawatts wrote: post #293
bbkane wrote: post #243

Earlier quoted context omitted.

Oh my goodness yes. I had the "great" idea to use Azure Functions to do a task at work. It's **ing insane how difficult it is to specify an Azure Function all in code with reasonable CI/CD, AD permissions, logging, and dev/prod instances. I wrote about what it takes at https://www.bbkane.com/blog/azure-functions-with-terraform/ but the experience really soured me on cloud services.

Is this... a deliberate attempt at constructing a Rube Goldberg machine? In all seriousness, at least 2/3rds of the complexity is because of your choice of tools and approach. Terraform alone makes things significantly more complex. If you just want to trigger a deployment, then a Template Spec made from a Bicep file could be banged out in like... an hour.[1] When in Rome, do as the Romans do. You basically took a Mi…

jiggawatts, this is an honest attempt. I'd LOVE it if there's an easier way I somehow missed.

And talk is cheap. I dare you to write a blog post or make a public GitHub repo doing the equivalent work (see Goals section) with your own tools. If you can, I'll be super impressed (not that my admiration is worth anything ).

One thing you'll run into is that AD roles and other authn aren't accessible via ARM templates/Bicep

Re: HN is up again

#297

Post by owl57 »

christophilus wrote: post #122
joshstrange wrote: post #34

I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.

Yep. Same. It says a lot about the quality of HN, I think. Also, I can't remember the last time it was down. For a while, I thought there must be something wrong with my internet connection or DNS config or something.

> For a while, I thought there must be something wrong with my internet connection or DNS config or something.

Definitely thought the same. Then I realized that I'm browsing trough work VPN and had a second thought: what if our admins decided to fight procrastination?

Re: HN is up again

#298

Post by greenie_beans »

cheesewhizemacs wrote: post #213
joshstrange wrote: post #34

I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.

> Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory. I do this too, and it's because this site is an addictive slot machine just like every other social networking site. I actually really hate this website, but I'm here almost every day, because I can't seem to break the habit. Neat. It's probably because I have a common impulse control…

i got addiction problems and this is the only website that's healthy for me. there's no endless scroll if you just visit the "news" and don't go to the "newest" page. i like how it's intentional to go to the next page. i usually only click through like ~5 pages at most, and once i've visited ~10-20 times in a day, most all the content is stale. it's also been helpful for me to set the procrastination limits. so many times i'll visit and can't scroll and just move on.

with that said, the comments are the most addictive part of this site.

Re: HN is up again

#299

Post by Ruphin »

rajamaka wrote: post #286
hooande wrote: post #285

Earlier quoted context omitted.

I bet thousands of people checked their network and DNS settings today. hn is what I use to make sure I have a connection

Same here, it used to be google but at some point it still just starting loading with no internet connection.

Easy to make sure you're not seeing some cached version by doing a search for random string from mashing your keyboard. These days literally any string of characters shorter than 10 will find some sort of result and there's very little chance you have it cached somehow.
Post reply on HN