Live data from Hacker News

“A Windows 7 deployment image was accidently sent to all Windows machines”

it.emory.edu

71–80 of 149 posts

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#71
post #48

Earlier quoted context omitted.

In my experience with campus networks, home directories are never stored locally on any remotely-administered machines. Any specially-configured researcher's machine that stored data locally would not have been subscribed to get the automatically deployed OS images.

>" As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted. " // If the SCCM server was pushed the "update" then there doesn't seem much hope for other machines? Surely no rule should be able to format the server running the ruleset; seems like a failsafe failure there at least.

None of the storage servers should have been storing the user data on the same volume as the OS the way a client machine would. So the network-mounted home directories should be intact and ready to use once the server OS is reinstalled. And while I don't know how SCCM works, I'd be surprised if this image push was affecting anything other than the primary physical drive (a wipe-all, populate-one recipe would be too obviously wrong and dangerous, right?).

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#75

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month ro…

>best minds of generation are deal with flaky VPN client

So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only solution). So much for Cisco...

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#78
Disasters as well as mistakes are unavoidable, such is life. A hallmark of a competent organization is how they handle the situation and recover from disasters or mistakes.

So far all the signs have indicated they are doing great in recovering. I just hope there won't be onerous processes and restriction afterward due to desire on "make sure it won't happen again" stance.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#79
post #15

I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7. This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.

Looks like it will be pretty easy these guys did it by accident.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#80
post #63
post #15

I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7. This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.

It must be reassuring that your new employer is so proactive in getting that XP upgrade project running.

Sometimes these products get bogged down because of three important problems outside of the control of IT. First off, you need to get in touch with all the upstream vendors to get updates to any sort of custom software that has compatibility problems with newer versions of windows; Vista/7 got a lot more strict about giving admin access, for example, which may cause problems in the updates. Second, you've got to keep in mind the training costs. There are a lot of users who may be brilliant financial minds that can make numbers dance and bow to their whims, but get terribly locked up if an icon changes. Doesn't make them horrible people by any means, but you've got to keep it in consideration when planning a rollout. Finally, you have to keep in mind the petty turf wars. If Joe in Accounting gets the upgrade to 7 before Bob in Legal, Bob in Legal may feel slighted and start raising a holy shitstorm, even if he's scheduled to be upgraded a week later. Upgrades are ugly, no matter when they happen. Sometimes that proactive upgrade project takes many years just because of all the moving parts involved.
Post reply on HN