dd if=outfile of=infile
Raw unadulterated fear followed by panic.
A full reinstall.
Triple checked dd params ever since.
Ask HN: What's the worst you've ever screwed up at work?
101–110 of 322 posts
Re: Ask HN: What's the worst you've ever screwed up at work?
#102I bet > 66% of these are something to do with databases. :-) My story (though I wasn't directly responsible): we were delivering our software to an obscure government agency. Based on our recommendation, they had ordered a couple of SGI boxes. I wrote the installation script, which copied stuff off the CD, etc. Being a tcsh afficianado, I decided to write it in tcsh with the shebang line #!/usr/local/bin/tcsh Anyways…
' And guess what's the value of "HOME" in bash?' In the rm line of the snippet above, "/some/location". Magic variables in bash tend to lose their magic once set.
nope. try it: set HOME; echo $HOME.
on IRIX, $HOME for root is almost certainly root. So the resulting command is rm -rf /
Re: Ask HN: What's the worst you've ever screwed up at work?
#103Boss: We have thousands of bad orders that must be fixed now! Me: No we don't. We have 121 bad orders. Boss: There are thousands of them! Me: No there aren't. There are exactly 121 of them. I'm sure. Boss: I'm not going to argue with you! Me: Good. Because you'd lose. I fixed 121 orders that night. The next day my login & password wouldn't work.
I might be dense, but I dont get it.
Re: Ask HN: What's the worst you've ever screwed up at work?
#104Lurker turned member to post this.
Second web related job at an insurance company, I was 20 years old at the time. We were heavy into online advertising, mostly banners at the time (this was right around when adwords started to get big). The company just bought out all of the MSN finance section of their site for the day-- it was a pretty big campaign ($100,000). We drove all the traffic to a landing page I had created with a short form to "Get a quote".
IT had given me permissions to push things live for quick fixes and such, I made a last minute design tweak and, you guessed it, broke something. I was checking click traffic and inbound leads and realized traffic was through the roof but leads were non-existent. This was about 45 minutes after the campaign was turned on. I jumped on the page and tested it out and got an error on submit. FUCK. I literally started to perspiration INSTANTLY.
Jumped into my form and quickly found the bug, can't recall what it was but something small and stupid, then pushed it live without telling a soul. Tested, worked, re-tested, worked. Ran some quick numbers to get a ballpark estimate on the damage I caused... several thousand.
Stood up and walked over to the two IT guys, mentioned I borked things and that I had fixed it... what should I do? I can still see the look on their faces. Shock, then smiles. Walked back to my desk and about 10 minutes later my two bosses show up (I worked for both dev & marketing managers).
They said thanks for catching the problem, not to worry. I did good for finding it myself, fixing it, and pushing it live. I was still sweating and shaking. They walk off and later that day marketing manager informs me MSN will refund us for the 45 minutes of clicks.
It took about a month before I felt competent enough to touch our forms again.
Re: Ask HN: What's the worst you've ever screwed up at work?
#105Earlier quoted context omitted.
What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complet…
What position are you starting from? Accurately assessing one's own competence is difficult and makes for boring reading, but since it's probably necessary here, I'll give some background. If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely. If they had dabbled with Debian, Fedora, SuSe or some…
You're welcome. If you want to deploy and maintain many machines, then maybe FAI[1] might be worth a look. It allows you to maintain a consistent state over an arbitrary number of machines running a Debian-based distribution, with _and without_ virtualization. We used it to run about 40 user-facing desktop machines and about the same number of cluster nodes. You basically have a central server that contains configuration, configuration-modifying scripts and package configurations. It is possible to define classes of machines, and one machine can belong to multiple classes, so you can have a part of the configuration identical on all machines and then other parts only on some of them.
Re: Ask HN: What's the worst you've ever screwed up at work?
#106I once worked for a company that schedules advertising before films. This wasn't in the US and the company had a monopoly over all of the ads shown across the country. It was my first programming job and done during university holidays, so I was there for a couple of months and then back to university. Toward the end of the following year I get a phone call: something was wrong with the system, it was allowing agents…
That time isn't normal where you are? How much did you make selling your system to the UK cinema industry? :)
I remember going to the cinema in the UK once in December, and being shocked by the ads. For a brief moment, I thought that I had been a victim of the tale.
Re: Ask HN: What's the worst you've ever screwed up at work?
#107This is about https://dictanote.co
I changed the login flow to use a different package. After pulling the latest changes in server, I restarted apache, opened the website to notice everything working smoothly and went to sleep.
9 hours later I wake up to check my inbox has 800+ emails. Django by default sends out email when an error occurs and a tiny mistake of not installing a package led to a lot of frustrated customers and well a huge pile of email in my inbox !
Moral of the story: Put that pip freeze > requirements.txt and pip install -r requirements.txt into your deployment flow.
Re: Ask HN: What's the worst you've ever screwed up at work?
#108My first real summer job was working for a computer store that also did tech support contracts with local businesses. I'll preface that the boss should never have given me the responsibilities he gave me, or should have gotten me to job shadow more experienced people, but the shop was tiny and I was actually the only full-time employee.
We had the tech support contract for the city's Mexican consulate. One of the things we were doing was patching and updating their server and installing a tape drive backup system. Server was NT4.
I'm in there doing work after 5pm, and wrongly assume that everyone's gone home for the day. Install some patches and the server asks me if I want to reboot. I say yes. Few moments later, a guy sticks his head into the server room and asks if I'd shut down or rebooted the server. Oh, whoops, someone's here. Yeah, I just installed some patches. Oh, OK, see ya.
Next day? Turns out he had been doing some work in their database where they track and manage visa applications. That database got corrupted when I did the server reboot while he was doing his work. That night, the backup process then overwrote the previous good copy database on the tape drive with the newly corrupted database. We had not yet started rolling over multiple tapes to prevent backups of corrupt data, though we were going to purchase some tapes for that purpose shortly.
Summer was ending, and I quit a week later to return to school. Horrible timing in terms of quitting! No idea what happened after that, as I was spending the summer in a city that was not my own. I do know that the original database developer contractor was on vacation at the time and so they couldn't reach him. I think the consulate was SOL. I regret rebooting that server without checking if anyone was working to this day.
Lesson learned? Don't assume anything when doing anything. Carried that lesson with me for the rest of my life. And find a boss who knows how to guide you if you don't have much experience in your area. I guess for founding startups, at least get an advisor.
edit: spelling
Re: Ask HN: What's the worst you've ever screwed up at work?
#109My first deploy at a once-top-10 photo hosting site as a developer was a change to how the DNS silo resolution worked.
Users were mapped into specific silos to separate out each level of the stack from CDN to storage to db. There was a bit of code executed at the beginning of each request that figured out if a request was on the proper subdomain for the resource being requested.
This was a feature that was always tricky to test, and when I joined the codebase didn't have any real automated tests at all. We were on a deploy schedule of every morning, first thing (or earlier, sometimes as early as 4am local time).
By the time the code made it out to all the servers, the ops team was calling frantically saying the power load on the strips and at the distribution point was near critical.
What happened: the code caused every user (well upwards of millions daily) to enter an infinite redirect, very quickly DoSing our servers. It took a second to realize where the problem was, but I quickly committed the fix and the issue was resolved.
Why it happened: a pretty simple string comparison was being done improperly, the fix was at most 1 line (I can't remember the exact fix). There was no automation, and testing it was difficult enough that we just didn't test it.
What I learned: If its complicated enough to not want to test using a browser, at least always build automation to test your assumptions. Or have some damn tests period. We built a procedure for testing those silos with a real browser as well.
I got a good bit of teasing for nearly burning down the datacenter on my very first code deploy, but ever since, its been assumed that if its your first deploy, you're going to break something. Its a rite of passage.
Re: Ask HN: What's the worst you've ever screwed up at work?
#110"Let go" a few hours into the first day on the job. A friend had referred me for a sysadmin job opening at a web hosting company in Florida. After a brief interview I got the job for a pretty decent salary and was told when I could start. What they hadn't told me was that my schedule would be tuesday to saturday. I had informed the hiring manager of my preferred schedule (monday-friday), but I guess nobody mentioned…
Which part was the screw-up?