Ask HN: What's the hardest problem you've ever solved?

How did the teacher not notice the headphones and question you about it during the exams?

Joining my current company and saving a relationship with a key client from the brink of termination.

The previous engineer on the account had overpromised and under delivered... in a rather big way. Not only that, but he also managed to misconfigure the enterprise automation software we use and accidentally delete the customer’s entire network share, then managed to get one of those cryptolocker viruses on the share after it was restored from backups, prompting a second restore. Regardless, the customer was pissed as hell and was on the verge of firing us.

After I was hired and put on the account (which I knew going in), I spent two days taking stock of all the shitty stuff the previous engineer had built, deciding what could be salvaged and what had to be discarded. Then, for the next two months, I went over the original contracts and built (or rebuilt) the originally promised system. I would basically work for 10 hours everyday, 7 am to 5 pm, then write an extremely detailed report to the customer explaining what I had done that way, which would take another 2 hours.

For the first week they ignored my reports. Then the CFO started taking interest, first by nitpicking, then asking questions, then praising. The rest of the customer’s team slowly came on board as well, especially once I started delivering the bits that were user-facing.

Today this customer is our flagship account, and has helped us land many other large accounts. But those two months were probably the most stressful of my life, not just because of the technical challenges but also due to having to simultaneously navigate the tricky political situation.

What a fun question :) And there are some very cool answers!

A few things that come to mind from my life, from back when I was still a kid/teenager:

- Building a three-way marble-track switch. There are many marble tracks that have these two-way switches that alternate between two tracks, like this one here: https://theworkbenchutah.files.wordpress.com/2013/04/06-marb... I wanted to build one that would evenly distribute marbles along three tracks. Took me quite a few prototypes until I finally had a working one. And all I had to work with was paper, tape, glue etc. No wood or metal, no bearings or precision mechanics, only a kid's craft stuff.

- A LEGO technics robot leg. This is a bit hard to explain without images. I wanted to build a legged robot (something spider-like) and needed legs that could move a foot both borward and back and also up and down. As the LEGO motors where quite big and heavy I didn't want to mount them on the leg itself (e.g. at the "knee") but have them all fixed inside the body and transmit their power via gears and shaft. The problem here was, that the up-and-down-motion had to go trough the forward-and-back-joint which meant that whenever the leg moved forwards or backwards it would also move a bit up or down, even if the up-and-down-motor didn't move at all. So I wanted to decouple these two motions. This would be trivial to do in software but I was just a kid playing with LEGOs so I wanted a mechanical solution. I managed it by using a mechanical differential to actually subtract the one motion of the other. Unfortunately, I then realised I didn't have enough motors for a full robot...

- Years later I was learning C++ and I wanted some kind of "linked variables" (I'm sure there's an official term for this): I.e. variables that would depend on others and would update whenever one of their dependencies changed. I though this would be a cool way of writing a GUI. With a lot of operator overloading and some abuse of the type system I could actually write things like

    x = a + b * 2;
    y = x / b;

And 'y' would update whenever a or b (or x) was changed.

I'm not sure if they really are the hardest problems I've ever solved, but it certainly felt so at the time ;)

I once wrote an app for Windows Mobile that implemented the L2CAP bluetooth protocol, and implemented the HID spec. This allowed phones to appear as bluetooth mouse/keyboards and be paired. It used the accelerometer and touchscreen to allow for cursor manipulation and had an onscreen keyboard for typing. Took me months to figure out the Bluetooth driver and to debug the HID issues, but when it worked, was pretty exciting. Just dug up an old analysis of it too https://forum.xda-developers.com/showthread.php?t=504730

The most elusive bug I ever investigated was also one of my first, in my first job after graduation, back in the early 90s.

TL;DR; spend months investigating a problem that didn't actually exist.

I was put on an investigation into why the data acquisition interface we were developing was not working. It was designed to plug into the serial bus on a Coast Guard icebreaker. Since we couldn't develop on the ship, one of the other engineers made an hour long recording of the bus traffic, which we were able to play back in a simulated setup in our lab.

Unfortunately, the traffic on the bus was not as reliable as we expected. There were supposed to be fixed number of data channels being broadcast every second. But frequently, seemingly at random, channels would disappear and then reappear a few seconds later. This made it impossible to identify the different channels, since they were order dependent. It usually wasn't even easy to tell which channel was missing. Usually multiple channels would missing at the same time. Our data readings would get corrupted and everything would fall apart. Since the end-product was to be a real-time engine diagnostics system, this was unacceptable.

When I was hired, the company had already been working at the issue for some time. I was tasked to try to find a pattern in the drop-outs, so that they could be predicted and our interface parsing adjusted as necessary. I tried to find correlations in the intervals between drop-outs, and the length of the drop-outs. I tried to apply smoothing functions so the effect of the drop-outs would be dampened. I tried doing predictive analysis on the data channels, so that when a drop-out occurred we would at least be able to make a pretty good guess which channel was missing, and realign all the other channels. Months went by. Nothing worked.

Then, one day while I was mulling over an printout of the data recording (literally 100s of pages of nothing but columns of numbers), I noticed something odd: almost every time a drop-out occurred, a channel would transition from positive to negative, or from negative to positive. That is, before the drop-out a channel would be positive, and after the drop-out had ended, the channel would be negative. Or vice versa. But a channel would never be 0. I poured over the entire recording, and there was not a single 0 value to be found.

The data recording had been made by plugging into the serial port on the ice breaker's data bus and using some modem software to record the traffic on the bus. We quickly put together a test setup using the same modem software, and sure enough, when a 0 value was received the software would not record it. It would just bloop right over it. There was nothing at all wrong with our data acquisition interface. The problem was with the recording we were using the test it. We had a new recording made using different modem software, and the problem was gone.