Live data from Hacker News

“A Windows 7 deployment image was accidently sent to all Windows machines”

it.emory.edu

121–130 of 149 posts

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#121

You know how in movies you need at least two people to bring their special secret keys, plug them in, and turn them at once to enable a self-destruct sequence? That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate. If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check…

In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.

It also helps to have sensible conventions for naming hosts and groups. If you need to select a subset of machines then you are sometimes going to make a mistake as simple is getting a wildcard pattern wrong. Instead have groups with explicit and obvious names that require no memory to understand.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#122
post #5

Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.

Indeed. Though the step described as "We are developing a more automated approach" is a bit scary. :)

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#123
post #74

Mistakes are made. In related news... Lawn care error kills most of Ohio college's grass http://www.wral.com/lawn-care-error-kills-most-of-ohio-colle...

I can understand the Embry problem a lot more simply because the distance between the decision and implementation of the decision is not that far.

How on earth do you "accidentally" load up enough weed killer to treat 54 acres of grass and never realize its the wrong stuff?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#125

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

Devops Borat: This not mistake. This how we test recovery operations in production.

This one goes on the first page of my playbook as of today.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#126

Earlier quoted context omitted.

Are you calling yourself one of the best minds of our generation?

Is it that surprising? I'd bet the average devops is in the top quartile, if not the top 10%.

Is that a statistics joke?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#127
post #124
post #94

Earlier quoted context omitted.

I thought Perl is what is holding the turtles together?

It's turtles.pl all the way down.

So, I feel like I might be being stupid and not getting something, but what is turtle? I can't find a programming language that seems to be related to it.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#128
post #124
post #94

Earlier quoted context omitted.

I thought Perl is what is holding the turtles together?

It's turtles.pl all the way down.

For sibling post:

"Turtles all the way down" is a "a jocular expression of the infinite regress problem in cosmology posed by the "unmoved mover" paradox."[1]

http://en.wikipedia.org/wiki/Turtles_all_the_way_down

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#129
post #124

Earlier quoted context omitted.

It's turtles.pl all the way down.

So, I feel like I might be being stupid and not getting something, but what is turtle? I can't find a programming language that seems to be related to it.

http://en.wikipedia.org/wiki/Turtles_all_the_way_down

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#130
post #120

Earlier quoted context omitted.

Are you calling yourself one of the best minds of our generation?

Hardly. No I meant on the receiving side of techs trying to fix VPNs.

Professionally I deal with much of the fallout from problems such as yours, and leading techs doing this kind of work. It really sucks, but for many problems like this the choice becomes spend-four-hours-reimaging-the-machine or spend-unknown-period-of-time-trying-to-fix-new-problem. The latter would be great if it was less than four hours, but it's often not, and until that time you / the user are without a machine.

After an hour or so of troubleshooting it's usually better to go with the reimaging, since all you / the user wants is to get back to working.

Ideally I try to get the entire broken machine captured and the user issued a new, fixed machine because then a fix can be developed and documented, but for those who end up in a new failure mode, it sucks. And with something like the Cisco VPN Agent? That's not uncommon at all...

Post reply on HN