Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

291–300 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#291
For those who are old enough to have used the original Visual Source Safe, they'll remember that "Get" and "Delete" were right next to each other in the context menu (edit: or at least very close by) of the code tree.

My first week at a game studio, I right-clicked the root of the tree and... yep... chose delete. And it "just worked".

Obviously not just my fault alone, but yeah, that was a pretty rough first week.

Re: Ask HN: Best “I brought down production” story?

#292
post #185
post #149

Earlier quoted context omitted.

A piece of the system (a junior developer) is allowed to make mistakes. The person responsible for architecting and protecting the system (the CTO)... less so.

That depends, this might have been this CTO’s first time as CTO. Without knowing the story, they could so have been pretty far out of their element and just lucky to be a founder or something. Even C-level people always have to have their first day as C-level and of course they will make mistakes. The important thing is learning from them of course.

> The important thing is learning from them of course.

Which is why I find it unacceptable that he repeated the mistake.

Re: Ask HN: Best “I brought down production” story?

#293
post #255

My first IT job was in a large call center. I was sweeping up in a data center and there was a keyboard cable stretched across a walkway. The keyboard wasn't movable (I don't remember why) so I unplugged it from the PC, swept around it, and plugged it back in. About 2 minutes later half a dozen people run into the room. Apparently the SUN workstation I had unplugged the keyboard from was a critical component of the c…

"there was a bug that forced a reboot when the keyboard was plugged in while the system was running."

That's not a bug, it's a feature! -- Sun

God, we fought with them about that. Even if it was a reasonable feature on a Sun 3 workstation, why put it on a high end server?

Re: Ask HN: Best “I brought down production” story?

#295

Earlier quoted context omitted.

Why did you write it Un*x? Is there a Unex or Unox? I've seen it written *nix to grab Linux and Unix.

That has precedent going way back, at least 34 years: https://unix.stackexchange.com/questions/2342/why-is-there-a... Doesn't explain why exactly the asterisk was put in that particular position. Maybe someone felt like it was odd to lead the word with an asterisk. :shrug:

The asterisk is there to avoid writing the word out in full, like "G*d" or "f*ck". There was a time when Unix[1][2] was a trademarked name and if you used it you had to attribute it. If you wanted to refer to the general family of Unix/POSIX/SysV/BSD/etc systems, you might be tempted to write "Unix"[1][2], but to avoid the presumably-Sauron[3]-like eye of the trademark holders, you'd bowdlerize it a little.

[1] Unix is a registered trademark of AT&T Bell Labs.

[2] Unix is a registered trademark of The Open Group.

[3] Sauron is a registered trademark of Tolkien Enterprises

Re: Ask HN: Best “I brought down production” story?

#296
I was part of a team that maintained end of day commodity trading software for a bank. Out of nowhere they got an $800 million inbalance one night. It was very odd and of such a magnitude that it was seen at board level (not good). We spent 12 nights watching end of days and debugging it in production before we found the issue.

It turned out that a new type of trade had been introduced which had a 0 days length. No trade we handled had ever had a zero day length; it was unheard of and the software did a divide by the trade length without checking for zero hence the bam. The div by zero exception was caught but only after it had skipped a Yen to $ fx conversion which is why the inbalance was so large.

Re: Ask HN: Best “I brought down production” story?

#297

Earlier quoted context omitted.

I was taught to also use a transaction and then check how many rows were affected before committing.

As was Tom Scott: https://youtu.be/X6NJkWbM1xk

I wonder if archive.org was around enough back then to capture any of it.

Re: Ask HN: Best “I brought down production” story?

#298

I went to a residential high school in a very rural state. Our internet went through a local college, then the state university. This is 1998. I was setting up my new main desktop with linux that year. When it got to network services, the installer asked me if I wanted dhcp and dns. "Of course I want dhcp and dns, I don't have a static IP here." It was asking if I wanted to install DHCP and DNS servers on this machin…

And thus DHCP snooping was born.

Re: Ask HN: Best “I brought down production” story?

#299

Earlier quoted context omitted.

After the bios screen screen would go black as it tries to load what I assume is the os. Then after a really long time an error message shows up, something like Acpi error, namespace lookup failure, Ae_not_found Then it goes back to bios and tries again. What I suspect happened is removing the nvidia driver led to some sort of circular dependency or lock on the system. This was when Ubuntu 20 first came out and offic…

Ahhhh, I see what you mean now. First of all, that tiny little detail about the ACPI error is actually incredibly helpful: it's one of the few messages that tend to still leak onto the screen when the system is configured to boot in quiet mode. Thus, Linux was actually partly booting 100% fine. If the system was then just automatically resetting after a bit, that definitely sounds like a driver fault, and if you were…

This is pretty amazing how you got all that from my simple hint. I also had a more detailed question regarding this on stack overflow but it seems to have been deleted from lack of response. I also have screenshots if you happen to be interested but by all means don't let me take up your time

Re: Ask HN: Best “I brought down production” story?

#300

Earlier quoted context omitted.

Ahhhh, I see what you mean now. First of all, that tiny little detail about the ACPI error is actually incredibly helpful: it's one of the few messages that tend to still leak onto the screen when the system is configured to boot in quiet mode. Thus, Linux was actually partly booting 100% fine. If the system was then just automatically resetting after a bit, that definitely sounds like a driver fault, and if you were…

This is pretty amazing how you got all that from my simple hint. I also had a more detailed question regarding this on stack overflow but it seems to have been deleted from lack of response. I also have screenshots if you happen to be interested but by all means don't let me take up your time

Feel free to email them over, sure. (Full disclosure, I'm sometimes terrible with reply latency.)

And FWIW, you actually tripped over a small longer-term cluster of internal grumbling - I like my screen to be completely blank at startup, and I've stared unimpressed at AE_NOT_FOUND-like messages (including that exact string) on my own system before I get to a login prompt.

So what kinda started out at "hmm, that's probably a misclassified priority or something..." ended up as a small wall of GitHub torvalds/linux links. Woops. (And then I didn't figure it out anyway... hmph)

Post reply on HN