Times I've Messed Up as a Developer

In my case, it was "Don't let anyone have you do testing/debugging on the production machine."

I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.

Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.

Well - at one point I cleared the wrong table - cleared the production and not the test table.

Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.

Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.

Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.

Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.

Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).