Ask HN: Great coding stories

We we're developing an Android app that involved a custom camera implementation. In the camera we we're getting the actual frames (about 15 fps was all we could manage on most Gingerbread devices) and saving them out in a custom file format.

We we're dealing with lots of device specific camera issues (HTC had one device with 2 back facing cameras that gave us a lot of trouble), and one day our QA person comes up to us and says: "It's working fine on this device, except it crashes when I try to film out the window."

We we're already feeling stressed about all the issues and didn't believe her. We figured it was just some race condition that it was a correlation != causation thing. We asked her to show us, and sure enough time after time you would turn the camera to face the window and it would crash.

We showed our manager and the designers, because we couldn't believe how weird it was.

The cause and fix turned out to be simple enough. We kept the lights off in the office for the most part, and the camera frame rate was jumping way up when the scene was bright. When the camera turned to the window, the fps went above what we had achieved before, and our app was throwing an OutOfMemoryException, since we weren't dealing with the frames fast enough, and they were stacking up.

It drove home the importance of real world testing to us.

I was managing a very busy Cassandra cluster of 14 nodes, these machines had 16GB of ram, and as a very horrible hack for OOM problems, had 8GB of swap space too. With Cassandra - you have to always have at least N megabytes of space free on the disk where N is the size of your largest table, this is because during "Major compactions" it's possible the table will have to be rewritten.

One of these machines was desperately running out of disk space, so I turned the swap space off to claim the extra 8GB (I just had to get the compaction out of the way, then these machines would be upgraded and this all wouldn't be a problem anymore)

So turning off the swap space, I could see the kernel moving data back into RAM, and I was also watching the diskspace fill up from the compaction. They were both going at a fairly linear rate, but the system was going to run out of disk space BEFORE the swapfile was released.

But not by much, the system would run out of diskspace about 30 seconds before the swapspace would be free --- now I know that this cassandra configuration was set so that a node wouldn't be considered "dead" unless it was out of communication with the cluster for longer than 10 seconds --- so I used KILL to freeze the cassandra process a few times, but never longer than 10 seconds.

I was able to freeze the process enough so that the swapspace was free'd before the diskspace ran out -- and the node communicated with the cluster enough to remain "active"

Lesson Learned: These machines were severely underspec'd!