271–280 of 424 posts
However, I think the CTO was the one who deserved to be fired.
Not having at the very least develop and production environments is a higher ups fault.
Where I work, developers can't even touch production systems, there's a separate team responsible for that.
I even have a solr, nginx, php, mysql, etc separate install of almost everything in my workstation, so I only touch test servers when doing testing.
Earlier quoted context omitted.
This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…
As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…
The bug is in the part that I though it was so obvious, I missed some check.
Earlier quoted context omitted.
Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"
This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…
This took down news information for a good chunk of Wellington finance for about half a day. (Fortunately Wellington, NZ is a tiny corner of the finance world).
Hilarious! But I admit I was super glad it was the boss playing chaos monkey, not me.
This happened to me once on a much smaller scale. Forgot the "where" clause on a DELETE statement. My screwup, obviously. We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use." (Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah!…
Personally I felt bad when I deleted some files, that were recovered within the hour, and learned from that experience. But when you create a monumental setback as the OP by simple mistake, that's an issue with people at higher ranks.
This happened to me once on a much smaller scale. Forgot the "where" clause on a DELETE statement. My screwup, obviously. We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use." (Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah!…
Who cancelled the backups? Why were they cancelled? Was it for non-payment of that service?
---
I worked for an absolutely terrible company as Director of IT. The CEO and CTO were clueless douchebags when it came to running a sound production operation.
The CTO would make patches to the production system on a REGULAR basis and break everything, with the statement "that's funny... that shouldn't have happened"
I had been pushing for dev|test|prod instances for a long time - and at first they appeared to be on-board.
When I put the budget and plan together, they scoffed at the cost, and reiterated how we needed to maintain better up-time metrics. Yet they refused to remove Dave's access to the production systems.
After a few more outages, and my very loud complaining to them that they were farking up the system by their inability to control access - I saw that they had been hunting for my replacement.
They were trying to blame me for the outages and ignoring their own operational faults.
I found another job and left - they offered me $5,000s to not disparage them after I left. I refused the money and told them to fark off. I was not going to lose professional credibility to their idiocy.
Worst company I have ever worked for.
More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.
The kind of table manipulations he mentions would be unspeakable in most companies. Someone changing the wrong table would be inevitable. If I were an auditor, I would rake them all over the coals.