Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

81–90 of 137 posts

Re: Times I've Messed Up as a Developer

#81

Fortune 500 company, Junior dev. Tried to log into our staging database, which has the same username, but different password than production. Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment. My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had l…

I have a hard time seeing how this was your fault. It seems like a poorly designed system is the root cause, not you.

Re: Times I've Messed Up as a Developer

#82
post #58

Late 90's, I'm lead dev for a large site. Company hires a DBA. We partner with a VERY large content provider. I'm awake for 48 hours buttoning things up, last minute stuff. I finally collapse from exhaustion only to be woken up at 7am on - go live day. Nothing works. A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as qu…

>We contact the DBA and explain the folly of his decision.

>He then proceeds to rename everything AGAIN on a live system.

What could his reasoning possibly be? Did he not understand your explanation?

Re: Times I've Messed Up as a Developer

#83
post #63

Earlier quoted context omitted.

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes. From man rm: -i prompt before every removal -I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes

This alias seems to be set by default in many linux distributions these days (and has been that way for many years now).

The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.

I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.

Re: Times I've Messed Up as a Developer

#84

Earlier quoted context omitted.

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

my go-to .bashrc hashes the hostname into a color and prints the prompt in that color. it has saved me from making a dumb mistake a few times.

Could you post the code somewhere?

Re: Times I've Messed Up as a Developer

#85
post #12
post #7

I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone! We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived…

Always write DELETEs as SELECTs, and once you run the SELECT and it returns the rows you want, then swap out the SELECT * with a DELETE.

I have a couple of SQL habits along these lines that I haven't seen others do.

If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:

  UPDATE items
  SET 
    active = 0,
    deleted_at = now() WHERE
  category = 'Foo'

  DELETE FROM items WHERE
  category = 'Foo' AND
  customer = 123
 
Yeah, it looks ugly, and I wouldn't use that formatting for "real code," with tests and whatnot. But for messing around in the query editor, I just find that I'm much more likely to accidentally leave off whole lines. I never accidentally select parts of lines. More generally, the idea is to make your partial query invalid syntax, if you leave off the last line.

I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:

  /*
  DELETE FROM items WHERE
  category = 'Foo'
  */
The idea here is just that I want to be required to select this text and run it. If I have some other queries in the window and somehow accidentally run all statements in the window (didn't select something I meant to or maybe a query is hiding beneath the scroll line), I don't want destructive queries to run.

EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.

Re: Times I've Messed Up as a Developer

#86
post #82
post #58

Late 90's, I'm lead dev for a large site. Company hires a DBA. We partner with a VERY large content provider. I'm awake for 48 hours buttoning things up, last minute stuff. I finally collapse from exhaustion only to be woken up at 7am on - go live day. Nothing works. A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as qu…

>We contact the DBA and explain the folly of his decision. >He then proceeds to rename everything AGAIN on a live system. What could his reasoning possibly be? Did he not understand your explanation?

incompetence

Re: Times I've Messed Up as a Developer

#87
On the first team I worked on at [corp] we had a corner of one of the bigger labs set up as a test environment, with a row of headless PCs that were used for simulating deployment scenarios. Sitting at the end of that row was a nondescript beige box that was barely distinguishable from the test boxes, except for being a bit older, that had our team's internal website, all of our documents, scratch file shares, databases, etc. on it. Of course there were no backups. At least two times while I was there, they hired temp workers and on day one gestured in the general direction of the shelf full of machines and told them something like, "OK, you can get up to speed by reimaging those boxes for this week's test pass..."

Re: Times I've Messed Up as a Developer

#88

Back in the nineties, at a previous employer, I wrote a code generator that read a kind of DSL from stdin and spat out C source to stdout. Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main sourc…

I've trained myself to always use >>.

Re: Times I've Messed Up as a Developer

#89
As for "Why So Many Dropped Production Databases?", you can always configure ssh with a combination of LocalCommand or use tmux to change the background of the terminal, when you are on a production server.

I did a blog post about it [1].

1: http://www.drinchev.com/blog/ssh-and-terminal-background/

Re: Times I've Messed Up as a Developer

#90
We were group debugging a crash that happened while using our API for a network device for NetWare under heavy load. We weren’t sure if the problem was our code or the IPX driver we had written for the particular NIC, so we were swapping in every NIC we had in the lab, and running some samples. Actually, I was doing the swapping. Turns out, the one time I forgot to power down was removing an NE-1000 (cheapest NIC available) and swapping in a 3Com 3C505, worth about $900 in 1988 dollars. The NE-1000 was fine, but the 3C505 was toasted. The PC was fine. The boss totally clowned me with some mean mugging, then smiled and said it was ok.
Post reply on HN