Ask HN: What's the hardest problem you've ever solved?

My problem was a lack of javascript logic firing and the answer was to wait for document load. Simple, I know, but I had nobody around me (physically) that could help me and explaining the problem to people in forums seemed impossibly abstract, primarily because I did not understand what the problem was. It was the context around my code that I had to fix, but I kept looking in the code itself.

That was probably one of my first big "ah ha!" moments and these moments are one of the reasons I still love programming. Tenacity, luck and skill became irrevocably connected that day.

I've solved many other problems over the years, far tougher than this one, but maybe never tougher for me in a relative scale. If I had never solved that problem, I sometimes believe my life path would have been totally different.

It was the early days of SOAP, and I had been assigned the task of integrating my employer's software with a third party's, so that the applications could share data. This third party org was a wealthy, powerful mega-corporation; and my employer was, well, not. The third party produced a spec for the interface, expected us to follow it, and offered no help from there.

I built a solution. It worked on my machine. Solved the problem. All was right in the world.

I moved it to the test environment. It worked again. Demoed it for one of our customers, and everyone was pleased.

Deployed it to our first beta tester. One lonely employee working accounts receivable, tucked away in the corner of our customer's office.

It crashed.

I checked everything. I mean everything. There are still particulars of that little Windows 2000 workstation that I can describe vividly. Which programs were installed, which patches were installed, how Windows had been configured, how the firewall worked, I even got permission to install a packet analyzer. My employer only had a handful of customers, and the beta test machine was near our offices, so I was over there personally a lot over the following weeks.

We brought in the customer's network support people. They found nothing. They could see the packets leaving, and an error coming back, but couldn't offer more than that.

We brought in the best networking engineer in my company. He was stumped.

What really shook my confidence was knowing that competitors mine had gotten this interface working. This wasn't some half baked project that I could blame on someone else. Others had succeeded where I'd failed.

I practically had to walk across broken glass to get on the phone with the third party's development team, but with enough pestering I pulled it off.

The phone call involved me sitting at the beta test workstation and firing off a request so that they could view it hitting their servers live. The developer who I spoke with immediately spotted the problem.

You see, when you send a SOAP request, you send the date and time that you're making the request along with it. The clocks on the client and the server were too far out of sync, my requests appeared to be coming from the future, and so the server disregarded them with a blunt error. Interestingly, the workstation clocks at my company's office weren't too far out of sync, which is why it worked in one place and not another.

Stuff I learnt:

1. Third party interfaces require a point of contact at both organisations who can talk with one another. This is non-negotiable.

2. If you send an error message that reads "Error", you're a bad developer and should return your computer science degree to your university and demand a refund.

3. No matter how well written the spec is, something always gets left out.

4. Persistence maters more than anything.

Turns out the bias current node (external RBIAS resistor sets bias current) for PCIe was routed too close to an inductor for a power rail. When the CPU was warm, the power rail pulled more current, causing the inductor to ring more, causing the crosstalk on the bias net to screw up the PCIe subsystem and hang the CPU.

Found the issue accidentally on a layout change. Had to prove it by drilling out the via and re-routing the signal with wire.