The tech world is rallying around a young developer who made a huge mistake

The original story (junior dev was given access to production / cleared the database while trying to setup his dev env) was funny to me. Actually, I can't really believe that what was described is possible. For me, there are two explanations:

- It's a prank by the OP. He posted a story out of his mind for whatever reason.

- It was all manipulated by a different employee of the company. He (by accident) broke the production db and decided to blame the junior dev: Most junior devs will just believe you (out of fear and inexperience) when you say to them that they broke the production.

The 2nd one seems more possible to me.

That still only applies to the second option. The first one (false story) is very much possible and it seems very plausible to me. It could be a bet, a sociological research or one of those "why not" ideas... we'll probably never know.

Unrelated to the story, but I've had a similar experience as to some of the stories described in the article - where a cascade of unseen problems landed on my lap and I took the blame even though little of it was my doing.

(Note, this was many years ago and mildly traumatic, so some details might be fuzzy)

In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients.

One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less).

So I come in. Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down. The client remotely shut down the servers in question but they didn't power off (missed clue). One by one I power them down, roll them out of the rack, insert memory.

This was a pretty routine request, so it went smoothly - so I had thought.

Switch the console to the database server, boot. The kernel starts without issue, then a slew of error messages. Too many to read.

The Sun 420R's were known to have memory riser issues, so I power off reseat the riser, and all the memory. Same problem.

I take out the new memory. Same problem.

Before I could troubleshoot the client guesses it's a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate.

Nothing comes up.

Not

One

System

So at this point the client starts to panic over the phone. They hang up on me.

So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages.

All the error messages are Bourne Shell (sh, not bash) errors.

Realizing this doesn't appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system. Everything looked great. So I tried to run a script, and then it became obvious - everything LOOKED too great.

All the init files had been reformatted. It said as much in a comment. A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server. The problem is, bourne shell didn't like some of the formatting changes - mainly space between variable, equals sign, and value declarations. Had he tested his changes, it would have been obvious what the problem was.

Testing my theory I fixed the formatting of a single script, and it immediately works without error. Problem found, now I need to fix all the scripts on all servers.

I call the client. No answer.

I call the client again. Answer. "Did you fix our servers yet?" "No, I found..." phone disconnects.

I call again, no answer.

I call again, no answer.

I call the NOC, asking they try to inform the client of the status.

So I proceed to start fixing the first server. It took an hour, but the server came up. The second server came up a little faster, and I proceeded to work on the third and fourth.

Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job.

I find this out as one of the analysts comes up from the NOC and tells me I'm relieved, and was asked to go home with no further information. So I explain what had happened and what I was doing to fix the issue.

The next day I am called into the office. Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client.

Happy birthday.

Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.

This is why, when you see gaping holes in production systems, security or otherwise, and those holes aren't your fault, your best course of action is to do nothing. If you explore it, you might trigger cascading effects and you'll be blamed for it. If you expose it, you'll embarrass someone potentially much more senior/powerful than you. If you do nothing, someone else falls in and you plead ignorance.

I've worked at two profitable software companies where every process was manual, and everyone had access to everything. At one, the production servers were just sort of sitting in ... not quite a hallway, but a random open space near the front door. I don't know what's hard to believe about this. Some companies put all their focus/investment on selling a product.