Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

281–290 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#281
post #265

Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics. At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it. After a bit of searching I f…

A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.

It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.

Re: Ask HN: Best “I brought down production” story?

#282
post #271
post #265

Earlier quoted context omitted.

A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.

I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.

This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?

Re: Ask HN: Best “I brought down production” story?

#283
post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

[deleted]

Re: Ask HN: Best “I brought down production” story?

#284
post #271

Earlier quoted context omitted.

I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.

This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?

When I read "wasn't entirely correct", and "probability of a crash was about one in a million", I wondered how you'd work that value out.

Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users.

I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.

Re: Ask HN: Best “I brought down production” story?

#285
post #265

Earlier quoted context omitted.

A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.

It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.

> The server lacked validation in this case and crashed when processing the invalid data.

As a backend dev, I'd point to this as the root cause, not the client sending invalid data.

Re: Ask HN: Best “I brought down production” story?

#286
post #285

Earlier quoted context omitted.

It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.

> The server lacked validation in this case and crashed when processing the invalid data. As a backend dev, I'd point to this as the root cause, not the client sending invalid data.

I see where you're coming from. I would say it's the cause of the crash but not the root cause of the issue as a whole. Both obviously need to be fixed.

Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future.

Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.

Re: Ask HN: Best “I brought down production” story?

#287

Earlier quoted context omitted.

Very curious what you mean by "bricked the hard disk"?

After the bios screen screen would go black as it tries to load what I assume is the os. Then after a really long time an error message shows up, something like Acpi error, namespace lookup failure, Ae_not_found Then it goes back to bios and tries again. What I suspect happened is removing the nvidia driver led to some sort of circular dependency or lock on the system. This was when Ubuntu 20 first came out and offic…

Ahhhh, I see what you mean now.

First of all, that tiny little detail about the ACPI error is actually incredibly helpful: it's one of the few messages that tend to still leak onto the screen when the system is configured to boot in quiet mode. Thus, Linux was actually partly booting 100% fine.

If the system was then just automatically resetting after a bit, that definitely sounds like a driver fault, and if you were still on the Ubuntu 18 drivers it sounds completely reasonable (for proprietary values of "reasonable" ._.) that you'd encounter a kernel panic or hardware lockup/reset or something like that.

--

I was curious why that ACPI error message leaked onto the screen, and presumed/guesstimated it was because it was being printed with a high log level/priority. I decided to go digging to see if my theory was correct.

Thanks for the verbatim quote, "namespace lookup" found the source of the message immediately: https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5.... So this uses acpi_os_printf() (defined at https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5...), a va_args thunk to acpi_os_vprintf() (defined immediately after), which... does a few things. It's honestly going to be shorter to just

  #ifdef ENABLE_DEBUGGER
    if (acpi_in_debugger) {
      kdb_printf("%s", buffer);
    } else {
      if (printk_get_level(buffer))
        printk("%s", buffer);
      else
        printk(KERN_CONT "%s", buffer);
    }
  #else
    if (acpi_debugger_write_log(buffer) 
there we go.

This is weird: it uses different paths if kdb support is compiled in. If it is, it'll only ever use printk() functions, but if it's not, it tries calling acpi_debugger_write_log() first and only does printk() things if that returns The printk_get_level() thing, added in 2016 (https://github.com/torvalds/linux/commit/abc4b9a53ea8153e0e0...), checks to see if the last line of text was a continuation line, and only starts a continuation line if the last line wasn't one. (Orthogonally relevant: https://lwn.net/Articles/732420/ coincidentally happened a year later)

I think that acpi_debugger_write_log() (https://github.com/torvalds/linux/blob/master/drivers/acpi/o...) is just a circular buffer sink. It dispatches via a function pointer to acpi_debugger.ops->write_log ("oh no, where does that go"); LXR to the rescue, which cross-references (https://lxr.missinglinkelectronics.com/linux/drivers/acpi/os...) (via the tiny usage link) to https://lxr.missinglinkelectronics.com/linux/drivers/acpi/ac..., which is... just a circular buffer writer (https://github.com/torvalds/linux/blob/master/drivers/acpi/a..., https://github.com/torvalds/linux/blob/master/drivers/acpi/a...). Huh.

If there's something spinning in the background continuously flushing the contents of the ACPI buffer to the screen, I have no idea how I'd surface that. But in terms of this particular call graph, I think the only potentially-interesting area is actually the KERN_CONT mechanism itself. I was fascinated to learn that the message prefix system actually works by writing { 0x01, } into the buffer (https://lxr.missinglinkelectronics.com/linux/include/linux/k...), where continuation lines are marked using "c". Interesting.

Now I'm wondering, if the last message to be printed to the console had a proper level and all, and the next line was a continuation line... what level does it get? I now see that the chances are this is not the reason why it leaks onto the screen, but that was actually my first thought.

I'm still learning/limping/stumbling through understanding all this, so this was just poking around for fun/practice :)

Practically speaking I do generally prefer to have an Absolutely Blank Screen™ while the system is booting, save for what I put on it :), and to that end the nuclear option is to add "fbcon=map:1", which basically reroutes the console to /dev/fb1 (assuming you don't have an fb1, aka a 2nd screen :) ) - but given that this literally gives you no console at all, yeah, not great for everyday usage (and unfortunately not great for many embedded scenarios where RS232 or network access would be trickier than just switching to a console on a ~VGA display). Hence my interest in seeing if it is in fact possible to squirrel away all the text, but still have functioning CTRL+ALT+F1 et al.

--

Also - when you're in the GRUB menu (which you can usually show by spamming ESC nonstop immediately after POST, if it doesn't automatically sit at the menu for a couple of seconds), hit 'e' to edit the selected item, identify in the wall of text the bit of the 'kernel' line that says "quiet" and/or "loglevel=..." and insert "loglevel=9 verbose debug", then hit CTRL+X to boot the modified entry. This can be made permanent by editing /etc/default/grub then DON'T FORGET :) to run `sudo update-grub` afterwards.

Generally you can effectively learn how to play with this in a VM, since once you're in GRUB most things are identical to real hardware. The majority of default configurations have like a 5-30 second timeout as well, so it's conveniently less necessary than it used to be to identify the exact nanosecond to start mashing the keyboard...

Re: Ask HN: Best “I brought down production” story?

#288
When I was in college and working on campus as a web developer I did a find-replace. I was in a hurry to get to my next class so I committed it in SVN and ran to class and muted my phone. When I went back to work a couple hours later I learned that my quick find-replace had taken down the entire campus email system. Apparently, I had broken something which caused automated error emails to be sent. However, I also learned that I had broken the method that sends the emails so it triggered a new error after sending the email and then tried to resend the original email. After a short period it had overloaded the campus email server. Thankfully, my boss was understanding but we had a nice long talk about testing code before committing it and not relying on automated tools like find-replace. There was also a new policy created about doing code reviews shortly after.

Re: Ask HN: Best “I brought down production” story?

#289
I had already resigned. First day of the last two weeks. I was updating a firewall rules in Cloudpassage. Their UI sucks, and doesn't make it clear what changes were made when you click "save". Apparently, I had accidentally changed port 443 from allow to block.

Well, when the site goes down the CEO charges into the DevOps eng room and starts screaming "we're under DDOS attack!" This was his go-to cause of any problem ever, and of course is never true.

Well anyway, with all the screaming and ruckus I had forgotten that I was just in Cloudpassage a few minutes prior changing something unrelated. So we are investigating the problem for almost an hour.

As services in AWS autoscale - they work for about 15 seconds, and then are no longer reachable. That's when I realize....once cloudpassage updates the firewall rules, the instance becomes blocked. Doh. I switched it back and everything went back to normal.

Two years without incident and then boom, ruined in the last two weeks. Felt awful, but fortunately they were cool about it.

Re: Ask HN: Best “I brought down production” story?

#290

2 days before I got married, I dropped the production database by accident from a GUI tool where “right-clicking” can be destructive if you click to fast. The application scheduled radio and television commercials and within 48 US states for a large international advertising group. The bigger problem was that the DBA had only been doing incremental backups and didn’t have a full back against which to run the incremen…

> a GUI tool where “right-clicking” can be destructive if you click to fast Fun fact, there’s a current bug in VS Code where right clicking on a file in the sidebar immediately triggers the menu entry that appears under the cursor, which happens to be delete. I’ve had it happen randomly for several months now.

Exactly the kind of thing that I am still afraid of to this day. :)
Post reply on HN