Times I've Messed Up as a Developer • Hacker News Board

Re: Times I've Messed Up as a Developer

#31

Post by swalsh » Tue, Oct 10, 2017, 3:55 PM UTC

donatj wrote: post #7

I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone! We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived…

I worked at a public company where write access to the production databases was required to go through an intranet site which logged the query, required a code review id, and limited the number of rows that could be deleted. It was more or less for sox compliance reasons, but it provided a nice bit of safety net around dangerous queries too.

It slowed things down, but the forced code reviewer process saved me once in a big way.

Re: Times I've Messed Up as a Developer

#32

Post by ge96 » Tue, Oct 10, 2017, 4:05 PM UTC

Tried to get around a firewall for a windows remote desktop, changed the port number through registry, logged out, could not log back in. Was not able to change it either, detached the AWS volume, attached it to another device, could not access the registry to change the port back... I could see the screenshot, machine is running but couldn't remote in.

Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.

Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.

Re: Times I've Messed Up as a Developer

#33

Post by Insanity » Tue, Oct 10, 2017, 4:15 PM UTC

We had weekly releases to production. during my first month I checked in some code without yet fully understanding the spaghetti'ness of the 20 year old codebase.

it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..

Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.

good bonding experience though. All had a laugh about it the next week. :-)

Re: Times I've Messed Up as a Developer

#34

Post by brianwawok » Tue, Oct 10, 2017, 4:18 PM UTC

charlesdm wrote: post #19

1ba9115454 wrote: post #11

Earlier quoted context omitted.

No one. It was lunchtime and it's possible no ne traded so far that day.

Well played

The flip side to the story..

what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?

Would be epic fired story.

I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.

Re: Times I've Messed Up as a Developer

#35

Post by avereveard » Tue, Oct 10, 2017, 4:19 PM UTC

1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on.

Since then I’ve adopted custom prompt and color scheme for production.

I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

Re: Times I've Messed Up as a Developer

#36

Post by brianwawok » Tue, Oct 10, 2017, 4:19 PM UTC

pavlov wrote: post #9

Harvey-Specter wrote: post #4

Earlier quoted context omitted.

> I only ever ssh into production servers on the white on red terminal so there's a strong visual signal This is a great idea.

I've found that quickly changing the color of Terminal windows is pretty much the only useful application for the MacBook Pro Touch Bar.

You have to back it into your profile. That way ANYONE that SSHes into that box gets the funny text colors... or at least a funny colored prompt.

Good way to save your bacon.

Re: Times I've Messed Up as a Developer

#37

Post by ChemicalWarfare » Tue, Oct 10, 2017, 4:20 PM UTC

Speaking of backups :) We had a very large mission-critical system that we developed and hosted for a fortune 50 client. Long story short - prod DB with tons of records took a crap, proper RCA was never done I suspected HW failure of some sort.

The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.

Re: Times I've Messed Up as a Developer

#38

Post by shagie » Tue, Oct 10, 2017, 4:21 PM UTC

kbart wrote: post #5

leejo wrote: post #3

I've seen a DBA drop a production table as well, fortunately it was a small in memory table for non persisting data that could be quickly repopulated. Still lead to about 5 mins of downtime. I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal…

"One is white text on black for dev work, the other is white text on red." Cool. Sorry, but I'm going to steal this idea from now on.

Also doable with desktops for those occasions where it is a remote screen share.

For UAT / Test environments that are different than dev, I use a yellow background.

Re: Times I've Messed Up as a Developer

#39

Post by brianwawok » Tue, Oct 10, 2017, 4:24 PM UTC

bg4 wrote: post #28

Stop mucking around in production. Stop. Seriously, stop.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

Re: Times I've Messed Up as a Developer

#40

Post by mikestew » Tue, Oct 10, 2017, 4:24 PM UTC

f4f4 wrote: post #27

Our 3-year-old production server hard drive, containing all assets and a database for multiple customers, died. No backups. For a few hours I was picturing not only the anger of our customers, but the demise of the whole company. Luckily the other drive in the RAID was OK and we didn't lose any data. Probably one of the stupidest, but also the most personally effective way to learn anything.

Account created an hour ago, and the only comment from this account is this one you’re looking at (which seems fine by my standards), and it’s insta-dead? I vouched for the comment, but am I missing something or is HN’s juvenile shadow ban broken again? (IP ban, maybe?)