Car allergic to vanilla ice cream (2000)

Dodge Ram Trucks had exactly the same problem. I also have no idea how two different companies with 100 years of history simply forgot how to paint cars.

The first was a VDSL connection I had at home. It worked great (fast, for the time) except when it didn't. It always failed in the evening. Techs would come out, bless it as being good, and leave -- because of course it worked while they were there. Unless they showed up and it was broken and then they'd declare that it was an outside problem, and that they'd have to get someone else to fix it (because the residential techs can't do overhead work).

I made lots of (very polite) phone calls, which results in more refunds and more service calls. More than once, my driveway and the street in front of my house looked like an AT&T convention.

This went on for months.

I had direct numbers and emails for tier 3 support and the local manager who oversaw this plant. We were all getting to know eachother too well, and there were boots on the ground addressing this problem as many as three times in week.

I eventually noticed that as the days got shorter so did the evening outages...and that if it was a cloudy day, then that day was often outage-free.

I had an epiphany: The problem might correlate with the angle of the sun, and the duration of exposure!

I checked my logs and the past weather, and sure enough: It lined up.

So I reported my findings, even though they seemed like nonsense as the words came out of my mouth, and they sent out some crazy-haired guy with bluejeans and an untucked shirt who was clearly not used to wearing a uniform, and who was also obviously not normally customer-facing.

"I know exactly why they can't find the problem," he said after I reiterated what I'd learned. "Your neighborhood still has old lead-sheathed overhead lines, and nobody knows how to work on that anymore."

"But I'm certified on that. I'm going to go back to the shop, pick up a bucket truck and get your line fixed. It will take me most of a day to do this, but I will be back when I'm done."

And it was getting pretty late, but he did come back to let me know that he found some things and fixed them. And I don't know what those things were, but it was fine after that -- and it stayed fine.

Thermal expansion letting cosmic rays leak into copper pairs wrapped in paper, tar, and lead? Who knows. I certainly don't know.

I've never encountered that stuff professionally (and it isn't your grandfather's 25-pair cable) and as this dude said, "nobody knows how to work on that anymore."

One day we started having flaky tests, seemingly out of nowhere. We quickly identified that the issue affected tests involving graphical X client applications, but then we struggled to make further progress. The issue was just impossible to reproduce in other conditions... Well, as it happens, the CI jobs were running on some desktop machines we had installed somewhere within our premises. It turned out that some gentleman had plugged a mouse into one of the machines, and left it lying around on the shelf. Since then, when one of the machines was under a heavy load, the fans would spin faster, causing more vibrations, in turn causing the mouse to move, ever so slightly. And for ungodly reasons, this had side effects on tests.

Fun fact: the machines were not on my site, I managed to diagnose this over SSH. I was quite proud :-)