Automation can also mean automated disaster.
[deleted]
“A Windows 7 deployment image was accidently sent to all Windows machines”
11–20 of 149 posts
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#12https://twitter.com/DEVOPS_BORAT/status/41587168870797312
"To make error is human. To propagate error to all server in automatic way is #devops."
Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#13This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work they'd done over the weekend (This was a weekend near the end of the semester when everyone was in crunch mode).
Long story short, the sysadmin was hired back and paid more than most of the profs. Academia may tend to skimp on salaries for certain positions, but sysadmins probably shouldn't be one of them.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#14[deleted]
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#15I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7.
This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#16Automation can also mean automated disaster.
Bigger tools have bigger sharper edges. It's important the message is stressed as the developers to operators worlds merge in the middle.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#17On the bright side they are no longer running XP!
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#18Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
It is impressive. I can't imagine dealing with that kind of nightmare.
I once worked as an admin on Solaris boxes at a big pharma company. There were ~77,000 users in their LDAP directory. I was very careful.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#19https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
Devops Borat: This not mistake. This how we test recovery operations in production.
Re: “A Windows 7 deployment image was accidently sent to all Windows machines”
#20Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
At one place I worked, in the days of XP, the 'index server'(?) had a problem and un-installed all the application programs. The basic OS was there but MS Office, MSIE, all the doodads just got removed as each machine logged in.
This was a small college, so the IT guys just went round explaining and told us to log out and log in again. Applications re-installed. No data loss and so no shouting.
Stuff happens. We did 'assignment action planning' that morning: mind maps, essay plans and research ideas. Results better than normal anyway