Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devops Horror Stories

statuspage.io

Post reply on HN
Print view View original

51–60 of 102 posts

Re: Devops Horror Stories

#51

Post by aidos »

jeffbarr wrote: post #48
rdw wrote: post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

There has never been an S3 limit. Some of our customers have millions of objects in a single bucket. When you do something like this you need to make sure that you have a good distribution of keys across the name space, and you need to think twice before you decide to write code to list the entire bucket. In most use cases at this scale, metadata and indexing are handled by something other than S3.

Guiltily admit to listing an entire bucket with 3,500,000 keys, nightly.

Re: Devops Horror Stories

#52

Post by Dylan16807 »

Fizzadar wrote: post #42

You can hardly be surprised when OVH or Hetzner go down, just consider the price. Putting every server in one location is just stupid... as always the best way to fight downtimes is to spread servers across multiple providers & DC's.

Going down is fine. Ignoring service problems is not fine.

Re: Devops Horror Stories

#54

Post by pygy_ »

0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is.

How did you locate it? Measuring ping latency from other machines?

Re: Devops Horror Stories

#55

Post by [deleted] »

zwp wrote: post #32
nailer wrote: post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

So tmpwatch traverses filesystems. Is that a bug? (thinks: "-xdev"). Edit: Reading tmpwatch.c: "Try hard not to go onto a different device". Perhaps old bug?

[deleted]

Re: Devops Horror Stories

#56

Post by Stealth- »

[deleted] wrote: post #55
zwp wrote: post #32

Earlier quoted context omitted.

So tmpwatch traverses filesystems. Is that a bug? (thinks: "-xdev"). Edit: Reading tmpwatch.c: "Try hard not to go onto a different device". Perhaps old bug?

[deleted]

He says a folder _under_ /tmp/

Re: Devops Horror Stories

#57

Post by cpt1138 »

(worst) update table set column = 'blah' WITHOUT a where clause (thank god for backups)

(2nd worst) delete from table where created Lesson learned, always backup and write the WHERE clause first

Re: Devops Horror Stories

#58

Post by walshemj »

jmccree wrote: post #43
ecopoesis wrote: post #24

Earlier quoted context omitted.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

I'd agree that in a perfect scenario, you should be able to push code at any time confidently. But for many companies and projects this is not really available. As well, in many organizations the person who has to fix broken stuff is not the same as who develops and pushes code. I'm not saying that's a good thing, but it is reality for many people. Even if it only happens once in your career, once you've had a dev pu…

Amen Brother

Its sad that most places dont have a proper technical copy (with a full copy of live data) to do full tests on TDD is all very well but you need to test the entire system.

Re: Devops Horror Stories

#59

Post by walshemj »

deckiedan wrote: post #30

Tape Archive System: write a tape, restore it again, and do MD5sum against the original data. Then we know it can be restored correctly, and the original data is deleted. Should be bullet proof? Alas, the 'write to tape' scripts I'd inherited didn't warn if they couldn't load a tape into the drive. There was a tape jammed in the drive, so the tape robot was refusing to load any new tapes, but kept on writing and rest…

I know one large company where contract operators managed to destroy every copy of a very large companys payroll by loading tape after tape onto a malfunctioning tape deck.

Re: Devops Horror Stories

#60

Post by nailer »

mhurron wrote: post #40
nailer wrote: post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

/mnt and /media exist for reasons. And root_squash and ... Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here. Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away. Thanks for the tmpwatch info btw.

Would have used /media but was thinking, if, say I forgot to unmount it or someone looked at a disk free or whatever, that it would be obvious that it was there temporarily.

Obviously that was incorrect, but the reasoning was, I think, sound.

Post reply on HN