21–30 of 103 posts
Many years ago, in my first tech job, we converted our typewriter-driven office to networked DOS PCs, running on Netware. One day, one of the legal secretaries, of self-described Roma ancestry, started complaining about her PC acting oddly...software not starting, spontaneous reboots ,etc. We replaced her machine to troubleshoot it, but the same thing happened with the replacement machine. We couldn't detect anything…
I can attest to this; we have a vacuum cleaning robot that has recently taken to spontaneous vacuuming at random times - the first time was naturally just after midnight and scared the life out of me :-)
I can attest to this; we have a vacuum cleaning robot that has recently taken to spontaneous vacuuming at random times - the first time was naturally just after midnight and scared the life out of me :-)
Doesn't even have to be digital. My furnace, with a very old, very analog thermostat, has been randomly turning itself on.
If it happens enough to be annoying, consider setting the T-stat to NOT call for heat and then go moving wires around until you find the place that has the intermittent short.
I wouldn't - the majority believes in an invisible sky friend.
I have a haunted TV: my 4 year old Sony Bravia (Android TV) reboots seemingly intentionally during the most interesting parts of TV broadcasts. And since a few weeks ago, it also regularly loses the Wi-Fi connection. Thanks Google and Sony!
Test failures that are flaky / sporadic often signal something you really need to look into - either you're not in control of your test's corner cases or your product's. Either way you can't ignore it.
But, sometimes, with the whole "running hundreds of thousands of tests of complex, interacting software on cloud infrastructure that has virtualisation, custom kernels, whatever ..." the answer is just "woooooo spooky". When it comes to customer boxes, it's even weirder ;-)
=== When PTRACE_SINGLESTEP got haunted ===
A few years ago, we saw a particular issue in which the PTRACE_SINGLESTEP operation (which should, you know, step by a single instruction!) would sometimes step two instructions - but it basically only happened in testing.
Eventually we figured out that:
* For a particular older enterprise distro on x86.
* If you tried to single-step a thread in one process.
* And a thread in another process hit a watchpoint at the same time.
* Then your thread would step by two instructions.
Spooky action at a distance. This might also have required you to be running on a virtualized system - it didn't matter to us at that point, since clearly we need to work around issues people may see on cloud infrastructure.
Of course, unless you tested debugger workloads extensively you'd probably never have two processes under debug simultaneously in order to notice this.
=== When transparent hugepages got haunted ===
When RHEL backported support for the Linux kernel's THP optimisation, quite a few years ago, to one of their enterprise kernels a bug also got backported that I'd call "extremely haunted".
THP worked fine, maps would automatically get huge pages where appropriate and would get split up again if necessary.
EXCEPT ... if a THP-ed memory location was:
* Write-protected because of COW memory sharing (e.g. after fork)
* And the page was split by a write
* And that write had come from a PTRACE_POKE operation, not an in-process write.
Then something crazy would go wrong in the system. That process, IIRC, would hang and become unkillable.
Some sort of goblin got into the memory management subsystem at that point and there was no getting it out again - gradually the whole system would start to lock up until it became unusable - eventually requiring the box to be reset. This happened even from an unprivileged process.
Again, this is something most workloads wouldn't see but - once we'd understood the underlying issue - we had to find ways to prevent it happening. You just can't trigger a kernel bug like that, even if you know it's not your fault. If you're always there when it happens, nobody is sympathetic to you!
Old tech? My current tech is possessed by those doing Satan's bidding. Constantly tempting me, always whispering in my ear to want more, making me covet what others have.