Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

241–250 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#241
post #166

Earlier quoted context omitted.

What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complet…

>If they were using Arch or Gentoo at home, If they are using gentoo, you should be finding someone else. Gentoo users are typically the most dangerous combination of profoundly ignorant, yet absurdly overconfident in their abilities. Seeing a bunch of autotools and gcc output scroll by does not teach you anything. But the mistaken reputation as an "advanced" distro makes people think that by using gentoo, they are t…

There's something to be said for the installer being a random liveCD and documentation for manually installing & configuring a system.

If you go though the handbook properly (and potentially enough times until you don't need it to install), then the amount of inherent linux usage and admin knowledge you can pick up is just phenomenal -for example I love the xkcd[1] even if it stopped applying when I started using gentoo.

I would expect a gentoo user to be comfortable on the command line, which doesn't hold true for a lot of other desktop users. That said, isn't it immense desktop linux has gotten to the point where the barriers to entry are grandma level low :)

It's also probably fair to say that every userbase has it's vocal idiots... [1] http://www.xkcd.com/1168/

Re: Ask HN: What's the worst you've ever screwed up at work?

#242
I built a content site, worked on it for two years and a few months. While updating the entire codebase to make the site faster and easier to work on for future updates, I accidentally deleted my database. 2 years gone, SEO traffic gone.

Takeaway: Sometimes, it takes a disaster to realize you were in another disaster anyways.

Re: Ask HN: What's the worst you've ever screwed up at work?

#243

My worst would have been catastrophic if I had waited one minute to make my mistake. I was commissioning a new control system at a power plant's water treatment facility. I was fairly new to the industry and had mostly looked over the guy who did the bulk of the work's shoulder as on the job training. This particular day the guy was out sick and we had to finalize a couple of things before we ran through the final te…

Rule #1 - Don't make things worse by guessing.

Re: Ask HN: What's the worst you've ever screwed up at work?

#244
post #129
post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

at my last job we had an unofficial slogan of "don't fuck with it friday"

Re: Ask HN: What's the worst you've ever screwed up at work?

#245

I was testing disaster recovery for the database cluster I was managing. Spun up new instances on AWS, pulled down production data, created various disasters, tested recovery. Surprisingly it all seemed to work well. These disaster recovery steps weren't heavily tested before. Brilliant! I went to shut down the AWS instances. Kill DB group. Wait. Wait... The DB group? Wasn't it DB-test group... I'd just killed all th…

I think you just started a new Agile-like trend:

'hyper-committed disaster recovery testing'

Re: Ask HN: What's the worst you've ever screwed up at work?

#246
Not me, but a co-worker at an internship I held:

Said person entered the number of metric tons of concrete 3 magnitudes higher than it should have been. Imagine the cost difference between 1.0 * 10^6 and 1.0 * 10^9 metric tons... Our boss was not pleased, to say the least.

But imagine how easy it is to enter a few extra zeros in an excel data cell. Yikes!

Re: Ask HN: What's the worst you've ever screwed up at work?

#248
post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Who packs their cell phone in a box when moving???

Who ships a product release while in no position to monitor the results?

Who ships a product release immediately after a commit?

Re: Ask HN: What's the worst you've ever screwed up at work?

#250
post #137

I've screwed up countless things, many much more expensive than this, but those stories aren't entirely mine to tell. But this was one of my first. Years ago, making boot floppies for a physics lab where I was reinstalling all the servers: I meant: dd if=/dev/zero of=/dev/fd0 I did: dd if=/dev/zero of=/def/hda Oops. Bye, partition table. (Always double-check everything you type as root.)

I meant to dd a partition to tape... and ended up doing the reverse on our fileserver. Tape is slow, so I caught it after a few kilobytes.

Somehow I managed to salvage the situation using the man pages for the filesystem, the C compiler, and making sure I did not reboot. Really don't remember much about that late night...

Post reply on HN