Ask HN: What's the hardest problem you've ever solved?

I had encountered some seriously incorrect outputs from the application server. The output in question was a function of internal states and current time (rounded to hours, it was kind of "hourly" display). The application server was set to log many input/output pairs so I was able to identify non-trivial amount of such errors, but I was unable to determine the cause. Common causes like memory corruption, time zones (as the business logic heavily depended on the local time), NTP synchronization and even the interpreter bug were considered and then rejected. Finally, after two weeks or some, I tried to simulate the function with varying current time and fixed internal states, and surprisingly a portion (but not all) of output from the past matched to the observed output!

It turned out that glibc `localtime` can misbehave in the way that it ignores the local timezone when it was unable to read `/etc/localtime`, and the Linux box the server was in had some issue on reading that (I never had fully identified it, this read was probably the only disk I/O from that server anyway). In lieu of this finding I have exhaustively and posthumously inspected the past logs; it was determined that the gross error rate was in the order of 10^-4 (!), and the way `localtime` used meant that the error can only alter a portion of the output. Studying the glibc code revealed that setting `TZ` environment variable would disable the UTC fallback, so I did so and the error was gone.

Lesson: Learn your moving parts, even if you don't know them in advance.

Years ago I came up with a simple equation for determining priority of software engineering bug fixes and small features:

Priority = (Benefit the feature provides to the product) / (Time to complete the feature)

where benefit is defined by the business side using any scale (say 1-100), and time to complete is defined by the assigned software engineer using any unit (perhaps man-hours). Regardless of what range the numbers fall in, 0 to 1 or 0 to 42, you end up with an ordered list of tasks which equally value business value alongside engineering time.

I came up with this while working at a medium sized company. I was frequently tasked with too many things to do. Despite tasks being organized in a Redmine-like tool, the implementation was still done in random order because nobody could define priority. This led to much miscommunication about what I was working on in the recent past and future. I used the equation to better communicate my activity and future plans with the business side. Given an ordered list of tasks from this equation, anyone could see clearly what was being worked on next.

The business side resisted attaching a numeric benefit to the features, presumably because that's hard. But it's equally hard to define the time to complete a software engineering task, and I eventually convinced them we needed to at least try to be scientific about both.

n.b.: I used this while working on a mature system. For a newer project or for tasks with more dependencies, it's probably still complicated to define priority. In the setting I was in, it worked great.

My boss's boss however thought it was condescending and nobody aside from myself ever made use of it. I hope to make use of it again one day, but after one bad experience with a medium sized company, I've stuck with smaller places where this is not as necessary.