This happened at one a former workplace too. Only the Solaris and Linux servers was untouched.
It "mildly" amused the *nix operations guys to see all the "point and click" colleagues panic.
“A Windows 7 deployment image was accidently sent to all Windows machines”
21–30 of 149 posts
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#22This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work…
[deleted]
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#23Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
100% agree. My initial response was to laugh, but dude, I so feel for the team that has to clean the mess up.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#24Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
It's not a standard across the board, but I've found academic institutions to be more honest about their technological mistakes (outside of large-scale breaches) than the private sector.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#25I've never been prouder of my alma mater. /s
You should be. Emory is the 2nd best school in Georgia! :)
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#26https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#27Isn't this more the fault of the system architect than the guy who accidentally fired the bad deploys?
It's similar to a database firehose: If you accidentally start deleting all data you should have a quick working backup ready to quickly bring the dead box up to production.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#28This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work…
The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#29[deleted]
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#30I was just watching the "What’s New with OS Deployment in Configuration Manager and the Microsoft Deployment Toolkit" session from TechED and hit the section on "check readiness" option which MS have added to SCCM 2012 in R2. It sounds like having this in part of the task sequence at Emory would have (at the very least) stopped this OS push from at least hosing all the servers.
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCI...