Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“A Windows 7 deployment image was accidently sent to all Windows machines”

it.emory.edu

Post reply on HN
Print view View original

131–140 of 149 posts

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#131

Post by Nexxxeh »

Havoc wrote: post #75
IgorPartola wrote: post #46

Earlier quoted context omitted.

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month ro…

>best minds of generation are deal with flaky VPN client So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only…

At the risk of asking a potentially dumb question, why can they only reimage the machine at head office?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#132

Post by jpwgarrison »

count wrote: post #129
crashandburn4 wrote: post #127

Earlier quoted context omitted.

So, I feel like I might be being stupid and not getting something, but what is turtle? I can't find a programming language that seems to be related to it.

http://en.wikipedia.org/wiki/Turtles_all_the_way_down

Count is correct, but I did have a flash to my childhood: http://en.wikipedia.org/wiki/Logo_(programming_language)

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#133

Post by dfc »

frogpelt wrote: post #123
8ig8 wrote: post #74

Mistakes are made. In related news... Lawn care error kills most of Ohio college's grass http://www.wral.com/lawn-care-error-kills-most-of-ohio-colle...

I can understand the Embry problem a lot more simply because the distance between the decision and implementation of the decision is not that far. How on earth do you "accidentally" load up enough weed killer to treat 54 acres of grass and never realize its the wrong stuff?

How do you click/not-click the button in SCCM...? How are these two events any different?

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#134

Post by drivingmenuts »

SoftwareMaven wrote: post #62
swasheck wrote: post #24

Earlier quoted context omitted.

It's not a standard across the board, but I've found academic institutions to be more honest about their technological mistakes (outside of large-scale breaches) than the private sector.

I worked at a company for several years that provided software to the IT groups in the higher ed vertical. What I found is there are two types of people in higher-ed IT (and they often congregate at different campuses): First are the people who believe in the mission. They are really good, and are willing to take a cut in pay for some combination of social good, great working environment, etc. These kinds of people t…

When I was in college, I worked for the IT department. While there's politics and bullshit no matter where you go, the politics and bullshit in Ed IT was not that noticeable where I was.

Furthermore, the profs were always happy to see me coming because I fixed their broken stuff without pointing any fingers.

It was heaven.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#135

Post by appplemac »

IgorPartola wrote: post #46
perlgeek wrote: post #12

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month ro…

In startup we gamify operations: 3 infrastructure failures, and ops team are out.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#136

Post by Havoc »

Nexxxeh wrote: post #131
Havoc wrote: post #75

Earlier quoted context omitted.

>best minds of generation are deal with flaky VPN client So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only…

At the risk of asking a potentially dumb question, why can they only reimage the machine at head office?

Not dumb at all. This is a professional service firm, so there is no real head office per se, but rather your "home office" - I just simplified it a bit for hn purposes.

Couple of reasons. Each country rolls their own custom image. Plus I need an office that has the encryption keys for the full disk encryption. Plus only 3 offices globally carry copies of my data (used when they can't pull the data off the hdd).

If I'm flying anyway I might as well go to home office - I know they have all the required stuff for my laptop.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#137

Post by Havoc »

c0nsumer wrote: post #130
Havoc wrote: post #120

Earlier quoted context omitted.

Hardly. No I meant on the receiving side of techs trying to fix VPNs.

Professionally I deal with much of the fallout from problems such as yours, and leading techs doing this kind of work. It really sucks, but for many problems like this the choice becomes spend-four-hours-reimaging-the-machine or spend-unknown-period-of-time-trying-to-fix-new-problem. The latter would be great if it was less than four hours, but it's often not, and until that time you / the user are without a machine.…

>spend-four-hours-reimaging-the-machine or spend-unknown-period-of-time-trying-to-fix-new-problem

Definitely. In our case its 8 hours minimum though for a re-image. Somehow the FDE makes pulling the old data off the machine slow.

You've got my sympathies though - I'd not like to be the one doing the IT in these cases. Can't be fun troubleshooting IT with that kind of time pressure.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#139

Post by commandar »

zachlipton wrote: post #101
k_sze wrote: post #85

Funny how they mention iTunes as one of the "key components" that are restored first, whereas Visio, Project, Adobe application are relegated to a second round.

Presumably iTunes is part of their base system image for all workstations, along with Office, Firefox, Adobe Reader, and the like. In other words, a basic set of software to handle standard officework tasks. iTunes is free and IT would probably rather distribute it everywhere than have people trying to install it themselves (or calling the helpdesk to get someone with administrator rights to do it). They then offer a…

Generally you're going to keep your base images in SCCM limited to software that's only infrequently updated. Otherwise somebody has to update the entire image every time an update gets pushed out. Instead, you package them up and deploy the apps on top of the base image at install time. It takes a little longer to deploy, but it takes less admin time to manage since the actual installs are automated anyway.

Re: “A Windows 7 deployment image was accidently sent to all Windows machines”

#140

Post by cfreeman »

harrystone wrote: post #18
miles wrote: post #5

Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.

It is impressive. I can't imagine dealing with that kind of nightmare. I once worked as an admin on Solaris boxes at a big pharma company. There were ~77,000 users in their LDAP directory. I was very careful.

That sounds terrifying.
Post reply on HN