Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

131–140 of 424 posts

Re: How I Fired Myself

#131

Post by pavel_lishin »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

> It's noble of the author to quit

It's not just noble. Considering how everyone treated him, and the company's attitude in general, he had no future there, and neither should anyone else.

Re: How I Fired Myself

#132

Post by MBCook »

mst wrote: post #119
MBCook wrote: post #79

Earlier quoted context omitted.

Ug. MyISAM. More than once (over the last few years) I've been doing some important update. I tend to do it the same way. START TRANSACTION; --run SQL-- --check results-- COMMIT; or ABORT TRANSACTION; Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup. So you always have to make a backup and check that the…

Any CREATE, ALTER or DROP on mysql does an implicit, silent COMMIT (not rollback) before running it. PostgreSQL can do DDL inside a transaction though.

Yep, that's another one to watch out for. But I'm used to that, it's when an update or delete that can't be reversed I get surprised.

Re: How I Fired Myself

#133

Post by [deleted] »

mst wrote: post #119
MBCook wrote: post #79

Earlier quoted context omitted.

Ug. MyISAM. More than once (over the last few years) I've been doing some important update. I tend to do it the same way. START TRANSACTION; --run SQL-- --check results-- COMMIT; or ABORT TRANSACTION; Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup. So you always have to make a backup and check that the…

Any CREATE, ALTER or DROP on mysql does an implicit, silent COMMIT (not rollback) before running it. PostgreSQL can do DDL inside a transaction though.

[deleted]

Re: How I Fired Myself

#134

Post by tlrobinson »

pmelendez wrote: post #23

There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident. 1) A cron job for the manually task you were doing. 2) Not working directly on production. 3) Having daily backups And this could happened to anybody. After midnight any of us are at junior level and very prone to do this…

Hopefully prioritized in reverse order

Re: How I Fired Myself

#135

Post by jpalacios »

clavalle wrote: post #85

Wow. So many mistakes. Working in production database? Bad. No backups of mission critical data? Super bad. Using a relational database as a flat data store? Super bad. Honestly...I think this company deserved what they got. Good thing the author got out of there. Hopefully in their new position they will learn better practices.

Yah this story made me cringe. What exactly do you mean by:

"Using a relational database as a flat data store? Super bad."

Are you referring to the users table? I am not too accustomed to using flat files, so I am curious.

Re: How I Fired Myself

#136

Post by developingJim »

Know a lot of others have said it, but no production backups? Blame a junior dev for a mistake that almost 100% of the people I've worked with have made at some point or another (including me)? I feel horrible for the author, it's sickening the way he was treated. Now they'll just move on, hire another junior, never mention this, and guess what? The next guy will do the same thing and there probably still aren't any backups. Didn't learn anything, well, other than how easy it is to blame one person for everyone's failure.

Re: How I Fired Myself

#137

Post by at-fates-hands »

mooreds wrote: post #100
liquidise wrote: post #98

Earlier quoted context omitted.

Exactly what you said. Rigor is truly the right word to use here. Cancelling your db backups is basically asking for a disaster. I'm not sure i have ever been at job that didnt require a db backup for some reason at some time.

To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?

This was my question. Usually the "junior" folks are shown how to do things by the senior engineers. The fact they threw this guy under the bus while letting the rest of the senior guys skate is appalling.

Part of your job as a senior developer is to ensure this very scenario doesn't happen, let alone to someone on your watch.

Re: How I Fired Myself

#138

Post by perlgeek »

xentronium wrote: post #16

This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible. Anecdote: I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important lat…

Reading those stories makes me realize how well thought-out the process at my work is:

We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the staging environment to have a rough upper bound for how long it will take, how much disc space it uses etc.

And we have a monthly sync from prod to staging, so the staging db isn't much smaller than prod db.

And the small team of developers occasionally decides to do a restore of the prod db in the development environment.

The downside is that we can't easily keep sensitive production data to find its way into the development environment.

Re: How I Fired Myself

#139

Post by stretchwithme »

While working for a large computer company in the late 90s, I joined a team that ran the company store on the web. The store used the company's own e-commerce system, which it was also selling.

The very first day, at home in the evening, I went to the production site to see if I could log in as root using the default password. Not a problem.

Anyone with any experience with the product could easily have deleted the entire database. I immediately changed the password and emailed the whole team.

No one ever responded.

Re: How I Fired Myself

#140

Post by paganel »

ritchiea wrote: post #106
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage".

Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger team-mates, a recent hire, "don't be afraid to break things!"

Post reply on HN