Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

271–280 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#271

Post by aragot »

mdisraeli wrote: post #254
hvass wrote: post #252

Earlier quoted context omitted.

There is a great lesson in one of pg's essays. If you push you better be there for few hours to monitor what is happening. You do not just push and leave. I hope we all learn from this thread.

In the IT services industry, we'll often talk about running a "war room" for the hours following a significant change, and then follow with a period of a nominated on-call person who has the authority to wake the project team up, tell them to drop everything and fix the issue. Funnily enough, change control actually runs a lot more smoothly in terms of getting past senior non-technical managers when you include eleme…

It sounds like you don't do continuous deployment. It divides the risk and improves agility. It pushes "change" to a non-management level.

Re: Ask HN: What's the worst you've ever screwed up at work?

#272

Post by alok-g »

10feet wrote: post #222
alok-g wrote: post #215

The following was not actually me, but worth sharing. They had ASIC design runs for research purposes once every three months, yielding your design on Silicon as ten 6" wafers. It gives enough parts for testing the first revision of your design. The person was carrying the wafers to a vendor for cutting into separate ICs and packaging or something. Gets to the parking lot, and where are the keys. Puts the wafers on t…

I don't think that is the lesson, I think the lesson is that it is clearly a two person job. One person to carry the wafers, the other person to remove and hazards/ open doors/ double check everything.

It's easy to say that after the fact. There are many delicate tasks people handle as a part of their day-to-day jobs, and not every task can afford more people to help without increasing the costs.

Re: Ask HN: What's the worst you've ever screwed up at work?

#273

Post by krishnasrinivas »

I had done "rm" of a big log file to free up space on the customer's server but our process kept on filling the log file. I assumed that the disk has got enough free space and I got busy with something else. The space was never freed up as the file descriptor was kept open by the process. Ultimately the entire disk got filled up by the opened log file and their server came to a grinding halt. I think the customer stopped using our product after that because we never heard from them again.

learning from this experience: never do an "rm" on the log file, instead do "truncate -s 0" on the log file.

Re: Ask HN: What's the worst you've ever screwed up at work?

#274

Post by embarrassed99 »

Leading a group working in an underground bunker on a live military radar site in the Australian outback, where it rains every few years. We had to open a rooftop cable duct and when the job ran overtime we closed it up with some rags that were to hand. That night it rained.

The next morning, the bunker was full to ground level and the automatic power cutoff had failed, as the float switch was directly under the cable duct and the water pressure of the deluge and kept the float depressed. By the time the water stopped flowing the float was under a foot of mud. The powered circuits were undergoing electrolysis and eating themselves away, made worse the the site managers refusing to drain the bunker or turn off the power until a week long arse-covering evaluation had been completed.

A few hundred million dollars of front line radar was out of action for several months.

Being a naive newly graduated engineer, I wrote a completely honest report and analysis. My boss said it was one of the best reports he had read and there was no impact on my career (if anything it got me noticed by the upper echelons of the organisation).

Lessons:

1. If you tell the truth you will be respected, even if it is incriminating.

2. If there is a way for something to go wrong it can do so (slight variation of Murphy's Law). Even if it's judged to be uneconomic to take preventative action, be aware of the possibilities, so you can make a conscious decision about the risk.

Re: Ask HN: What's the worst you've ever screwed up at work?

#275

Post by roryokane »

preinheimer wrote: post #120

I ended up as the architect for a new live show we were putting on. You could either pre-purchase some number of minutes, or pay per minute, it was like $4.99/minute or something insane. The billing specs kept changing, as did the specs for the show itself. New price points, more plans, change the show interface, add another option here, etc. The plan had been to do a free preview show the day before to work out the…

Your “duplicate billing code” strategy is called N-version programming (https://en.wikipedia.org/wiki/N-version_programming).

Re: Ask HN: What's the worst you've ever screwed up at work?

#278

Post by kchoudhu »

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

Ah, memories.

I did this in a production database (thought it was a QA environment) and brought trading on the mortgage desk of an investment bank to a grinding halt on September 14th, 2008.

The DBAs saved my 23 year old ass that day. I make it a point to send them beer on 9/14 every year.

Re: Ask HN: What's the worst you've ever screwed up at work?

#279

Post by laxk »

slipangel wrote: post #192

sudo chown -R myname:myname / Learned: Learning on the job as you hack away on problems is great, but recognize that it's one part enthusiasm and one part risk management. Also learned to never try anything on the command line that wouldn't want to see pulled from my bash history and stuck on the breakroom fridge. Also learned to cope with humiliation well.

Put space in front of any command to avoid history.

Re: Ask HN: What's the worst you've ever screwed up at work?

#280

Post by lambdaphage »

jboggan wrote: post #207

I love these topics. ~ 2007, working in a large bioinformatics group with our own very powerful cluster, mainly used for protein folding. Example job: fold every protein from a predicted coding region in a given genome. I was mostly doing graph analysis on metabolic and genetic networks though, and writing everything in Perl. I had a research deadline coming up in a month, but I was also about to go on a hunting trip…

Nice. I once needed to do reciprocal blast for the complete genomes of about 300 bacterial species. That's on the order of half a billion queries, but the work was embarrassingly parallel, and each discrete job only took about 90 seconds. I wrote a little shell script to kick them off on the cluster, and went home.

I woke up the next morning to several inbox screens' worth of messages from angry people I didn't know, demanding explanations for what I did to their jobs and their cluster. I don't think I have ever biked to the lab faster.

After multiple rounds of palm-drenching emails with the cluster sysadmins and the computational mathematics group PI (and my own boss agonizingly cc'ed), we determined the cause. The cluster sysadmins, lacking imagination for the destructive naivete of their users, had not foreseen that anyone would want to submit more than 10^4 jobs at once. That broke the scheduler, preventing other people from running jobs and me from canceling them. Meanwhile the blast jobs blew past the disk quota, leading to a Hellerian impasse where I somehow lacked the space to delete files so I could create space. I still don't fully understand it.

I believe it took a full day to get the cluster back online.

Post reply on HN