We accidentally re-imaged all of the Windows servers with Linux the other day. Nobody noticed though...
“A Windows 7 deployment image was accidently sent to all Windows machines”
41–50 of 149 posts
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#42https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
I think it does happen often but isn't as well reported. I certainly know of more than one place that's suffered from this kind of accident (thankfully not places I personally work so I've not had to deal with the fallout. These are places I have friends or family who work there)
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#43https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
The most frightening movie ever for sysadmins -- The Sorcerer's Apprentice.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#44https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
"In startup we are practice Outage Driven Infrastructure."
I'm not sure that's so unlike Netflix's Chaos Monkey.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#45Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?
Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.
Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#46https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully:
"In devops we have best minds of generation are deal with flaky VPN client."
"Single point of failure in private cloud is of usually Unix guy with neckbeard."
These are gold.
Edit: based on the above advice I once grew out a neckbeard while going through a multi-month rollout of a large product. It itched like crazy, but I did work much faster to get rid of it.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#47> A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks. Wow. That is very unfortunate, to say the least...
> As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.
We apologize again for the fault in the subtitles. Those responsible for sacking the people who have just been sacked have been sacked
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#48Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?
In my experience with campus networks, home directories are never stored locally on any remotely-administered machines. Any specially-configured researcher's machine that stored data locally would not have been subscribed to get the automatically deployed OS images.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#49This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work…
The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?
[deleted]
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#50We accidentally re-imaged all of the Windows servers with Linux the other day. Nobody noticed though...
That probably means nobody is using the servers you maintain and your department should be eliminated completely.