Earlier quoted context omitted.
Now there are already three comments adding no value.
4.
make that 5
How I Fired Myself
401–410 of 424 posts
Re: How I Fired Myself
#402Earlier quoted context omitted.
Honestly, you don't. The IT engineering in power and other SCADA systems is downright scary.
sounds like an opportunity to me.
It's a hard space to break into. The businesses are conservative about any change, and new vendors are fairly un-trusted. Further, it is a bit different of a world than normal software realms, due to crazy long legacy lifetimes. Another problem is that the money isn't as big as you'd think, its a surprisingly small field.
All that being said, there is opportunity, just not easy opportunity. And a huge number of the people in it are boomers, so there is going to be big shake-ups in the next decade or two.
Re: How I Fired Myself
#403Earlier quoted context omitted.
"walking into your closet and pulling a cable" is not free, if your planned disaster recovery is not a seamless failover, but a process to recover data with some work and limited (nonzero) downtime/cost to business. For example, our recovery plan for a financial mainframe in case of most major disasters was to restore the daily backup to off-site hardware identical to production hw; however, the (expensive) hardware…
To paraphrase: if you go with a awful hack job for your disaster recovery plan, testing is more expensive. And to extend: you won't actually test because it's "too expensive", and your disaster recovery plan won't work. How is this distinct from "Don't hack your systems to make them work. Absolutely do hack at them to test."? I don't see it. This just sounds like "my business doesn't have the financial capacity to en…
We did know that we can recover backups because we did it for small parts of data, and we know that we can do disaster recovery because (a) we did test this, though very rarely; and (b) we had successfully recovered from actual full-scale disasters twice over ~7 years.
But successful, efficient disaster recovery plan doesn't always mean "no damage" - it often means damage mitigation; i.e., we can fix this with available resources while meeting our legal obligations so that our customers don't suffer; not that there aren't consequences at all - valid data recovery plans ensure that data recovery really is possible and details how it happens, but that recovery can be expensive. And while you can plan, document, train and test activities like "those 100 people will do X; and those 10 sales reps will call the involved customers and give them $X credit", you really don't want to put the plan into action without a damn good reason.
For example, a recovery plan for a bunch of disasters that are likely to cut all data lines from a remote branch to HQ involves documenting, printing & verifying a large pile of deal documents of the day, having them shipped physically and handled by a designated unit in the HQ. The process has been tested both as a practice and in real historical events.
However, if you "pull a wire in the closet" and cause this to happen just so, then you've just 'gifted' a lot of people a full night of emergency overtime work, and deserve a kick in the face.
Re: How I Fired Myself
#404Earlier quoted context omitted.
To paraphrase: if you go with a awful hack job for your disaster recovery plan, testing is more expensive. And to extend: you won't actually test because it's "too expensive", and your disaster recovery plan won't work. How is this distinct from "Don't hack your systems to make them work. Absolutely do hack at them to test."? I don't see it. This just sounds like "my business doesn't have the financial capacity to en…
We did know that we can recover backups because we did it for small parts of data, and we know that we can do disaster recovery because (a) we did test this, though very rarely; and (b) we had successfully recovered from actual full-scale disasters twice over ~7 years. But successful, efficient disaster recovery plan doesn't always mean "no damage" - it often means damage mitigation; i.e., we can fix this with availa…
OK, tough love time:
All I can say is that you're very lucky to have a working system (and probably a company to work for), and I'm very lucky not to work where you do. Seriously, your "test" of a full disaster recovery was an actual disaster! More than one!
And frankly, if your response to the idea of implementing dynamic failure testing is that someone doing that should be "kicked in the face" (seriously, wtf? even the image is just evil), then shame on you. That's just way beyond "mistaken engineering practice" and well on the way to "Kafkaesque caricature of a bad IT department". Yikes.
Admittedly: you have existing constraints that make moving in the right direction expensive and painful. But rather than admit that you have a fragile system that you can't afford to engineer properly you flame on the internet against people who, quite frankly, do know how to do this properly. Stop.
Re: How I Fired Myself
#405Earlier quoted context omitted.
Your comment makes me think. Are you implying that this is a good practice? I mean, in fact I do something similar. At our company also a lot of stuff goes wrong. Somehow it surprises me that there was no major fuckup yet. But I do realize that I need to watch out all times that blame never concentrates on me. It is so easy to blame individuals, it just suffices to have participated somehow in a task that fucked up.…
You (and the other commenters with similar strategies) are wasting productive years of your life at jobs like these. You should go on a serious job hunt for a new position, and leave these toxic wastelands before they permanently affect your ability to work in a good environment.
As soon as you find an environment where no one ever plays the blame game, let me know.
Re: How I Fired Myself
#406Earlier quoted context omitted.
Neither of those things would have fixed this problem. 1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids" 2. Transaction complete. ... ... "oops!"
Why type "TRUNCATE TABLE Raids" every time? Type it once, test it, save as script, schedule that script.
That is definitely another bug in this company's process. This should have been automated. It's much harder to screw up running a script!
Re: How I Fired Myself
#407Earlier quoted context omitted.
Hindsight is 20/20.
If you have only one copy of data, especially if it is important, the chance of something happening to that copy, either hardware, software, or human error, is always big enough to justify a backup. No hindsight needed for that.
Thinking about this just makes my mind go numb. All someone had to do was have the idea that they should back their database up. It would be done in 5 minutes, tops!
I automated ours on S3 with email notifications in under an hour...
Re: How I Fired Myself
#408Earlier quoted context omitted.
I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...
We are all born naked, bloody, and screaming; the only thing we know is how to work a nipple. Everything else has to be learned. One of Toyota's mantras is "If the student has failed to learn, the teacher has failed to teach." Their point is that managers are responsible for solving issues that come from employee ignorance, not line workers.
Babies have to learn how to work a nipple, it's not some innate human knowledge.
Re: How I Fired Myself
#409More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.
More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…
Just think, the company spent millions of dollars training this guy in protecting production data, and then instead of treating him like gold, and putting him in charge of fixing all the weaknesses that allowed this to happen, they pushed him out of the company. Stupidity beyond measure.
Not surprised that this was a gaming company because lack of teamwork seems to be endemic in that industry.
Re: How I Fired Myself
#410While i fully agree with the position of author not being responsible entirely, i find it hard to believe it happened the way it appears to be.
It could be but there are bunch of loopholes. - I can believe that he was lousy enough to click no "delete" on users table. I can believe that he when the dialog box asked "are you sure you want to drop this table" he clicked yes. I can believe that after deleting he "committed" the transaction. But what i can't believe that the database let him delete a table which was base for every other table implemented by a Foriegn key constraint ? It could be argued that due to efficiency they hadn't put constraints on the table but it's hard to digest.
Probably the story is some what tailored to fit to a post.