Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics. At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it. After a bit of searching I f…
A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.
Ask HN: Best “I brought down production” story?
281–290 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#282Earlier quoted context omitted.
A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.
I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.
Re: Ask HN: Best “I brought down production” story?
#283Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
Re: Ask HN: Best “I brought down production” story?
#284Earlier quoted context omitted.
I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length. AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.
This seems like a very specific guess and it's not related to the actual issue. I wonder how you arrived at this idea?
Predictable statistics screams RNG, to me. Especially if the code author knew it was an imperfect solution, and didn't know of one that wouldn't cause this crash. (I've done stuff like this before.) Also, for the client to crash the server, it'd need to give it some form of bad/confusing data, but for some reason this doesn't happen with fewer users.
I think that was my logic at the time of writing that comment, though I've thought of another few guesses while typing this out/reading your follow-up.
Re: Ask HN: Best “I brought down production” story?
#285Earlier quoted context omitted.
A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.
It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
As a backend dev, I'd point to this as the root cause, not the client sending invalid data.
Re: Ask HN: Best “I brought down production” story?
#286Earlier quoted context omitted.
It's a slight simplification. Technically a client side bug lead to bad state which was then reported to the server. The server lacked validation in this case and crashed when processing the invalid data.
> The server lacked validation in this case and crashed when processing the invalid data. As a backend dev, I'd point to this as the root cause, not the client sending invalid data.
Fixing only the client side issue removes this crash entirely, but leaves one open for similar issues in the future.
Fixing the server side issue fixes the crash and related ones, but doesn't address the root cause of the faulty data being generated by the client.
Re: Ask HN: Best “I brought down production” story?
#287Earlier quoted context omitted.
Very curious what you mean by "bricked the hard disk"?
After the bios screen screen would go black as it tries to load what I assume is the os. Then after a really long time an error message shows up, something like Acpi error, namespace lookup failure, Ae_not_found Then it goes back to bios and tries again. What I suspect happened is removing the nvidia driver led to some sort of circular dependency or lock on the system. This was when Ubuntu 20 first came out and offic…
First of all, that tiny little detail about the ACPI error is actually incredibly helpful: it's one of the few messages that tend to still leak onto the screen when the system is configured to boot in quiet mode. Thus, Linux was actually partly booting 100% fine.
If the system was then just automatically resetting after a bit, that definitely sounds like a driver fault, and if you were still on the Ubuntu 18 drivers it sounds completely reasonable (for proprietary values of "reasonable" ._.) that you'd encounter a kernel panic or hardware lockup/reset or something like that.
--
I was curious why that ACPI error message leaked onto the screen, and presumed/guesstimated it was because it was being printed with a high log level/priority. I decided to go digging to see if my theory was correct.
Thanks for the verbatim quote, "namespace lookup" found the source of the message immediately: https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5.... So this uses acpi_os_printf() (defined at https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5...), a va_args thunk to acpi_os_vprintf() (defined immediately after), which... does a few things. It's honestly going to be shorter to just
#ifdef ENABLE_DEBUGGER
if (acpi_in_debugger) {
kdb_printf("%s", buffer);
} else {
if (printk_get_level(buffer))
printk("%s", buffer);
else
printk(KERN_CONT "%s", buffer);
}
#else
if (acpi_debugger_write_log(buffer)
there we go.This is weird: it uses different paths if kdb support is compiled in. If it is, it'll only ever use printk() functions, but if it's not, it tries calling acpi_debugger_write_log() first and only does printk() things if that returns The printk_get_level() thing, added in 2016 (https://github.com/torvalds/linux/commit/abc4b9a53ea8153e0e0...), checks to see if the last line of text was a continuation line, and only starts a continuation line if the last line wasn't one. (Orthogonally relevant: https://lwn.net/Articles/732420/ coincidentally happened a year later)
I think that acpi_debugger_write_log() (https://github.com/torvalds/linux/blob/master/drivers/acpi/o...) is just a circular buffer sink. It dispatches via a function pointer to acpi_debugger.ops->write_log ("oh no, where does that go"); LXR to the rescue, which cross-references (https://lxr.missinglinkelectronics.com/linux/drivers/acpi/os...) (via the tiny usage link) to https://lxr.missinglinkelectronics.com/linux/drivers/acpi/ac..., which is... just a circular buffer writer (https://github.com/torvalds/linux/blob/master/drivers/acpi/a..., https://github.com/torvalds/linux/blob/master/drivers/acpi/a...). Huh.
If there's something spinning in the background continuously flushing the contents of the ACPI buffer to the screen, I have no idea how I'd surface that. But in terms of this particular call graph, I think the only potentially-interesting area is actually the KERN_CONT mechanism itself. I was fascinated to learn that the message prefix system actually works by writing { 0x01, } into the buffer (https://lxr.missinglinkelectronics.com/linux/include/linux/k...), where continuation lines are marked using "c". Interesting.
Now I'm wondering, if the last message to be printed to the console had a proper level and all, and the next line was a continuation line... what level does it get? I now see that the chances are this is not the reason why it leaks onto the screen, but that was actually my first thought.
I'm still learning/limping/stumbling through understanding all this, so this was just poking around for fun/practice :)
Practically speaking I do generally prefer to have an Absolutely Blank Screen™ while the system is booting, save for what I put on it :), and to that end the nuclear option is to add "fbcon=map:1", which basically reroutes the console to /dev/fb1 (assuming you don't have an fb1, aka a 2nd screen :) ) - but given that this literally gives you no console at all, yeah, not great for everyday usage (and unfortunately not great for many embedded scenarios where RS232 or network access would be trickier than just switching to a console on a ~VGA display). Hence my interest in seeing if it is in fact possible to squirrel away all the text, but still have functioning CTRL+ALT+F1 et al.
--
Also - when you're in the GRUB menu (which you can usually show by spamming ESC nonstop immediately after POST, if it doesn't automatically sit at the menu for a couple of seconds), hit 'e' to edit the selected item, identify in the wall of text the bit of the 'kernel' line that says "quiet" and/or "loglevel=..." and insert "loglevel=9 verbose debug", then hit CTRL+X to boot the modified entry. This can be made permanent by editing /etc/default/grub then DON'T FORGET :) to run `sudo update-grub` afterwards.
Generally you can effectively learn how to play with this in a VM, since once you're in GRUB most things are identical to real hardware. The majority of default configurations have like a 5-30 second timeout as well, so it's conveniently less necessary than it used to be to identify the exact nanosecond to start mashing the keyboard...
Re: Ask HN: Best “I brought down production” story?
#288Re: Ask HN: Best “I brought down production” story?
#289Well, when the site goes down the CEO charges into the DevOps eng room and starts screaming "we're under DDOS attack!" This was his go-to cause of any problem ever, and of course is never true.
Well anyway, with all the screaming and ruckus I had forgotten that I was just in Cloudpassage a few minutes prior changing something unrelated. So we are investigating the problem for almost an hour.
As services in AWS autoscale - they work for about 15 seconds, and then are no longer reachable. That's when I realize....once cloudpassage updates the firewall rules, the instance becomes blocked. Doh. I switched it back and everything went back to normal.
Two years without incident and then boom, ruined in the last two weeks. Felt awful, but fortunately they were cool about it.
Re: Ask HN: Best “I brought down production” story?
#2902 days before I got married, I dropped the production database by accident from a GUI tool where “right-clicking” can be destructive if you click to fast. The application scheduled radio and television commercials and within 48 US states for a large international advertising group. The bigger problem was that the DBA had only been doing incremental backups and didn’t have a full back against which to run the incremen…
> a GUI tool where “right-clicking” can be destructive if you click to fast Fun fact, there’s a current bug in VS Code where right clicking on a file in the sidebar immediately triggers the menu entry that appears under the cursor, which happens to be delete. I’ve had it happen randomly for several months now.