Live data from Hacker News

“A Windows 7 deployment image was accidently sent to all Windows machines”

it.emory.edu

101–110 of 149 posts

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#101
post #85

Funny how they mention iTunes as one of the "key components" that are restored first, whereas Visio, Project, Adobe application are relegated to a second round.

Presumably iTunes is part of their base system image for all workstations, along with Office, Firefox, Adobe Reader, and the like. In other words, a basic set of software to handle standard officework tasks. iTunes is free and IT would probably rather distribute it everywhere than have people trying to install it themselves (or calling the helpdesk to get someone with administrator rights to do it). They then offer additional applications on an as-needed basis to individuals and departments with specific tasks. So the designers who do print publications and the faculty who teach digital art might get the Adobe suite, while people in Facilities who plan construction will get Project. This keeps licensing costs down and simplifies systems according to their uses.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#102
post #63

Earlier quoted context omitted.

It must be reassuring that your new employer is so proactive in getting that XP upgrade project running.

Sometimes these products get bogged down because of three important problems outside of the control of IT. First off, you need to get in touch with all the upstream vendors to get updates to any sort of custom software that has compatibility problems with newer versions of windows; Vista/7 got a lot more strict about giving admin access, for example, which may cause problems in the updates. Second, you've got to keep…

I think you missed the joke.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#103
hah! delighted to see this here.

my roommate works at the emory library and has had a fun slow week there of coming home early many days because no one could do work. they were apparently also given laptops as an interim solution, but those somehow also wiped themselves eventually (?).

poor IT people...just as they're starting to get a handle on the actual sitation it starts blowing up on the internet.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#104
post #75

Earlier quoted context omitted.

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month ro…

>best minds of generation are deal with flaky VPN client So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only…

Are you calling yourself one of the best minds of our generation?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#105

You know how in movies you need at least two people to bring their special secret keys, plug them in, and turn them at once to enable a self-destruct sequence? That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate. If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check…

In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.

Event that is hardly enough. There should be a physical (well virtual physical) obstacle towards launching a high stakes command.

The system should be able to assess the scope of a task, and ask you to confirm 10 times if it has to, in blinking red dialogs, to make sure you really want to do what you are doing.

Of course, it's crucial that "clicking 10 times" is not the default behavior for any trivial action. Or boredom and the subsequently formed mechanical 10-click habit of the operator will kill the effectiveness of this approach...

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#106

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

>Frankly, I'm surprised things like this don't happen more often.

They do. This happened to the largest bank in Australia mid 2012[1]. Very similar circumstances. I've been told that SCCM's UI doesn't help here- something about the default action when nothing is selected to apply it to all devices managed by SCCM. Someone more familiar with SCCM may want to correct me here.

[1] http://delimiter.com.au/2012/07/30/disastrous-patch-cripples...

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#107
post #5

Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.

I imagine this is what the sysadmins conversations at Emory are like right now: https://www.youtube.com/watch?v=eOkAyUmyQko

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#108

Earlier quoted context omitted.

In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.

But then that leads to people automatically clicking "Yes" to the "Are you sure..." dialog. Though even I would pause at "Are you sure you want to reformat 12,500 machines including this one?" ;-)

Having people type in some sort of auto-generated confirmation key might give them pause, seeing as the technique is seldom used.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#109

Earlier quoted context omitted.

Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.

Yes, there are a lot of options, both commercial and non-commercial, for full drive backups (then you restore the using those and the incrementals). Do that for your provisioning servers and you can redeploy a lot of the infrastructure based on that.

I see, Thank you. Somehow I was under the impression that automating and scaling the tedious work of analyzing and restoring boot sectors and the like couldn't be done. I suppose that it's easier to plan for though if you restore a whole drive from a backup image rather than restoring a random set of files by going on a sector hunt ?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#110

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month ro…

Same for TheCodelessCode. A lot of these are cryptic and weird, but some are pure gold. Especially since no one understands the koans until they fall flat on their face just like the student does and a huge floodlight turns on.
Post reply on HN