51–60 of 149 posts
Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
In my days of university tech support.
Earlier quoted context omitted.
Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.
Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.
It was both direct and funny enough that I was only mildly annoyed that the cluster was down.
Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?
In my experience with campus networks, home directories are never stored locally on any remotely-administered machines. Any specially-configured researcher's machine that stored data locally would not have been subscribed to get the automatically deployed OS images.
If the SCCM server was pushed the "update" then there doesn't seem much hope for other machines? Surely no rule should be able to format the server running the ruleset; seems like a failsafe failure there at least.
Earlier quoted context omitted.
Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.
Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.
That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate.
If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check email" and "open facebook".
Same way it shouldn't be easy for users to delete or corrupt their data by accident due to some omnipotent action innocently shoved right in between other trivial actions.
I wouldn't blame the person who triggered this re-imaging process. I'd blame those who designed the re-imaging interface, to allow it to happen so easily by accident.
https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.