system("tar --delete-files -czf archive.tar.gz $datadir/"); What could possibly go wrong?
$datadir could be emtpy
Ask HN: What's the worst you've ever screwed up at work?
291–300 of 322 posts
Re: Ask HN: What's the worst you've ever screwed up at work?
#292I unknowingly reset serial number counters in a bicycle part's database, so now there are a few hundred people in the world with high end bike hubs that overlap each other. Lesson: Keep the code that touches production databases as simple as possible so it's easy to verify exactly what it does. I was using a framework's database tooling incorrectly because I never dreamed what I used would touch the databases's count…
Chris King?
Re: Ask HN: What's the worst you've ever screwed up at work?
#293I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…
> I've only cried literal tears once in the last ten years, over business.
> Don't code when you're distracted.
Same story here, I can't remember the exact scenario but I was concurrently acting under all three of my titles (Developer, Architect, Escalation Engineer/Critical Debugger). The customer (who was 7 hours different to us) had been battling for 3 months and we were getting nowhere (all thanks to a, since fixed, bug in WinDBG which essentially came down to broken stack traces in certain scenarios), for those 3 months I had been working 20 hour shifts (development by day, support by night).
Under that strain I eventually made a screw up with the dev and it cost QA time. The MD of my region had a sit down with me and I
> cried literal tears
Needless to say they were impressed that I someone cared so much and sent me home to sleep. The next day I came in and decided to go through the 800kloc codebase line-by-line and see what could be causing the issue - I found it in a few hours.
Lesson #4: Get sleep.
Re: Ask HN: What's the worst you've ever screwed up at work?
#294[deleted]
Virtual goods? Fired why? Ambien is known to do these kinds of things. Was it because of the aderall and ambien combined made them thing you were a risk due to drug abuse? I don't get the firing over your description of the situation to be honest.
Virtual goods worth real money I assume.
Re: Ask HN: What's the worst you've ever screwed up at work?
#295mysql> drop database PRODUCTION Do you really want to drop the 'PRODUCTION' database [y/N] y^Hn Database "PRODUCTION" dropped
The correct answer is always ^C
Re: Ask HN: What's the worst you've ever screwed up at work?
#296Happened to a colleague: it was the end of the day, and we were packing up to leave. He used Ubuntu on his notebook, so he typed "shutdown -h now" on his shell prior to closing the lid. Seconds later he's groaning, having noticed it was a SSH session to the production server... It wouldn't be a big deal, wasn't for the fact it was an EC2 instance, and back then halting the instance was equivalent to deleting it perma…
I once did a "shutdown -h now" on a remote server on a customer site with nobody there who knew anything about the server when I'd meant to do a reboot....
Re: Ask HN: What's the worst you've ever screwed up at work?
#297One time I tried to change a column name in a production database. I learned that when you change a column name, mysql doesn't just change a string somewhere, it creates a new table and copies all the values from the old table into the new one and when that table has millions of rows in it, it really slows down your production server.
I keep thinking that's the most ridiculous thing ever.
Re: Ask HN: What's the worst you've ever screwed up at work?
#298way back in the late 90s stone age of interactive ad agencies, we were doing our first really big gig for hp. it was a demo shipping out to retail stores showcasing one of their products -- a run of 30,000 stamped cd roms. i was the one developing the macromedia director app running on the cd. we were on-time. we were ready to send them out the door. it was awesome. and then we tested the rom outside of our network..…
Hey m3mnoch, you asked me to remind you to tell us about the time you ran a database query on production that nuked the entire website for the publicly-traded software company which relied on -- wait for it -- the website to do all its commerce!
Re: Ask HN: What's the worst you've ever screwed up at work?
#299Earlier quoted context omitted.
All true, but that still doesn't answer what possesses someone to pack their cell phone in a box .
Seriously guys, take it easy. My question is: Who never makes mistakes? I certainly do not belong in that set. And I like Patrick's candid business anecdotes. Snarky comments like this might disencourage people from writing useful advice here. Which would be very unfortunate.
Just trying to be light-hearted about it, sorry if that did not come across.
Re: Ask HN: What's the worst you've ever screwed up at work?
#300Earlier quoted context omitted.
In the IT services industry, we'll often talk about running a "war room" for the hours following a significant change, and then follow with a period of a nominated on-call person who has the authority to wake the project team up, tell them to drop everything and fix the issue. Funnily enough, change control actually runs a lot more smoothly in terms of getting past senior non-technical managers when you include eleme…
It sounds like you don't do continuous deployment. It divides the risk and improves agility. It pushes "change" to a non-management level.
"continuous deployment" is not a panacea to the issue. There's two primary ways of doing it, from what I understand:
continuous deployment to a dev/test environment is the easiest for most organisations to move to. Due to the live environment being mission-critical, they can't afford the risk of any degradation of service. So you push regularly (after passing test suites) to a test environment, get a small subset of users working, and at some point then push out to live from that. But I suspect that isn't what you are referring to, as this is too similar to typical change management.
the alternative definition of continuous deployment is that of constantly pushing to live, initially for a subset of users then rolling it out from there gradually, but always on the live environment. In many large, often 'cloud-y' solutions, that subset might be all users. Except you can't have a public transport ticketing system fail at peak times. A hospital patient record system must stay available for staff, and give sufficient notice before any possible impact to service to allow manual processes to be used. Payroll, accounts, HR systems... all of these have failure modes that have financial penalties at best.
Hence continuous deployment only makes sense when you can afford to risk service, possibly with significant impact.