31–40 of 149 posts
https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work…
The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?
Disclaimer: dev at a university.
One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement.
There was a bug in the program, and the edits were sent down as loads.
A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items.
Barry said it was the first time he got to meet the president of the company that day.
One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement.
There was a bug in the program, and the edits were sent down as loads.
A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items.
Barry said it was the first time he got to meet the president of the company that day.
This was back when phone lines were used to transmit data, and it took several hours just to transmit an item file. So, the store locations were out of business until the item file could be restored.
Isn't this more the fault of the system architect than the guy who accidentally fired the bad deploys? It's similar to a database firehose: If you accidentally start deleting all data you should have a quick working backup ready to quickly bring the dead box up to production.
Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?
https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
Bad stuff doesn't just happen to Windows networks.
Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?