Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#61

Post by simcop2387 »

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

Yep, I typically do SUPDATE just for fun that way. And only do that AFTER building the where clause with a SELECT * FROM foo WHERE ... so that I always start with the clause when making the update. Might be paranoid but it always seems to work out for me that way.

Re: Ask HN: What's the worst you've ever screwed up at work?

#62

Post by xauronx »

cgh wrote: post #19

I was in a remote meeting and failed to realise my laptop's camera was broadcasting. A roomful of people saw me, clad in horrid workout clothes, jam my finger up my itchy nose and scratch my balls. Key takeaway: always check the cam.

And make sure your phone is muted. The first conference call is easy. When you have one every day for a year and it becomes so common place... sometimes you forget. I've heard some people on my team coughing obnoxiously, yell at people driving, doing the dishes, etc. Mute your shit, and tell your team mates immediately when they aren't muted.

Re: Ask HN: What's the worst you've ever screwed up at work?

#63

Post by dllthomas »

Robin_Message wrote: post #56
dllthomas wrote: post #51

Earlier quoted context omitted.

' And guess what's the value of "HOME" in bash?' In the rm line of the snippet above, "/some/location". Magic variables in bash tend to lose their magic once set.

I assume `set HOME = /some/location` is the tcsh syntax to set a variable. In Bash, it doesn't do anything useful.

Ah, right, duh.

Re: Ask HN: What's the worst you've ever screwed up at work?

#64

Post by michaelmior »

gargarplex wrote: post #27
michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I always add a LIMIT even when not necessary. Why doesn't MySQL have a version control baked in? Even if it preserves just the last n hours of state..

That's what backups are for. With appropriate checkpoints, you can roll back to any point in time with a reasonable level of precision.

Re: Ask HN: What's the worst you've ever screwed up at work?

#65

Post by [deleted] »

michaelochurch wrote: post #17

Tried to prevent a massive product failure. It failed anyway, but I wasn't around when it did and there would have been no "I told you so" credit even if I were. One of those "big company" lessons, but probably applicable to startups (which have an even higher ego density).

[deleted]

Re: Ask HN: What's the worst you've ever screwed up at work?

#67

Post by krisdol »

earino wrote: post #15

easy: me: "unix definitely won't just let me cat /dev/urandom > /dev/sda" other: "sure it will" me: what I learned? unix will absolutely let you hang yourself. 1998, production server for a fortune 5 company.

Why in heaven's name did you try that in production?

Re: Ask HN: What's the worst you've ever screwed up at work?

#69

Post by sillysaurus2 »

yan wrote: post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows

I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system?

In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or what topics should I study? I want to become an expert at modern sysops. Modern deployment, hardening, backup, managing dozens of boxen, etc.

I've been thinking of going through any MIT OCW on the subject, but it seems like hard-earned experience might not necessarily translate well to an academic setting. What would you recommend I do?

Post reply on HN