Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sysadmin mistakes start-ups make

cloudkick.com

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Sysadmin mistakes start-ups make

#71

Post by SwellJoe »

tptacek wrote: post #69
SwellJoe wrote: post #35

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

Or, you can skip the O'Reilly book --- and most of the mistake you could make with the simplest BIND setup --- and just use djbdns.

While I respect your opinion on most topics, I believe you're way out of your element here. I'm talking about fundamental misunderstandings of the protocol, not stupid typos in the configuration file: Glue records wrong at the registrar, incorrect NS records, non-existent MX records, misunderstanding of propagation/caching/TTL, no PTR, etc. These are the kinds of problems I see all the time.

djbdns doesn't make the person doing the configuration understand those things any better than BIND does.

Re: Sysadmin mistakes start-ups make

#72

Post by DTrejo »

blasdel wrote: post #44
michaelbuckbee wrote: post #26

Earlier quoted context omitted.

That isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?

It's a terrible idea to mix and match drive models, because they'll have drastically different performance characteristics, and if you're lucky you'll only get a little worse than the lowest common denominator on all axes. What quality OEMs do is make sure they never ship you drives from the same manufacturing batch in one enclosure.

How do you make sure to buy from a quality OEM?

Re: Sysadmin mistakes start-ups make

#73

Post by azim »

gstar wrote: post #68
jrockway wrote: post #66

Earlier quoted context omitted.

10 million per day is about 100 per second. SQLite performs about this quickly. I wrote a test script and it did 125 (unindexed) lookups per second. Then I ran two of these tests at the same time, and the rate stayed about the same. I have 8 cores, so I made 8 processes, and it was the same. 125 requests/second * 8 * 86400 seconds/day = 86_400_000 requests per day. I added another thread writing as quickly as possibl…

That -is- fast, but I still have trouble reconciling that deep down in my computer, a human readable SQL query gets built, and then another process parses that SQL. Seems so wasteful building and then parsing a human readable string for something that's happening on the same machine. I know nothing of SQLites's internals, but wouldnt it make more sense to parse the query once and then store a compiled version of the…

Yes, This is known as a prepared statement. You compile a parametrized statement once, then execute it as many times as you like with different arguments.

Also, SQLite, unlike most other databases, is an embedded database which does everything in-process rather than invoking multiple processes.

Re: Sysadmin mistakes start-ups make

#74

Post by jlc »

omouse wrote: post #51
SwellJoe wrote: post #35

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

Could you name that book? Do you have know of any other books people should read for sys-adminning?

I like the Evi Nemeth book for admin:

http://www.amazon.com/UNIX-System-Administration-Handbook-3r...

Re: Sysadmin mistakes start-ups make

#76

Post by tptacek »

SwellJoe wrote: post #71
tptacek wrote: post #69

Earlier quoted context omitted.

Or, you can skip the O'Reilly book --- and most of the mistake you could make with the simplest BIND setup --- and just use djbdns.

While I respect your opinion on most topics, I believe you're way out of your element here. I'm talking about fundamental misunderstandings of the protocol, not stupid typos in the configuration file: Glue records wrong at the registrar, incorrect NS records, non-existent MX records, misunderstanding of propagation/caching/TTL, no PTR, etc. These are the kinds of problems I see all the time. djbdns doesn't make the p…

I'm pretty sure somewhere in my resume there's something relevant to this conversation, but I can't remember.

Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS config issues, like matching PTR records, or setting up reasonable TTLs and SOA field values. It's right there in the data format.

That's apart from where the dots and semicolons go in the database files. Which yeah is a mistake I managed to make a lot, even after writing a dynamic DNS server for our ISP, and is a problem that djbdns makes go away completely.

Sorry. You're going to have to add this to the list of topics I'm insufferable about. ;)

Re: Sysadmin mistakes start-ups make

#77

Post by SwellJoe »

tptacek wrote: post #76
SwellJoe wrote: post #71

Earlier quoted context omitted.

While I respect your opinion on most topics, I believe you're way out of your element here. I'm talking about fundamental misunderstandings of the protocol, not stupid typos in the configuration file: Glue records wrong at the registrar, incorrect NS records, non-existent MX records, misunderstanding of propagation/caching/TTL, no PTR, etc. These are the kinds of problems I see all the time. djbdns doesn't make the p…

I'm pretty sure somewhere in my resume there's something relevant to this conversation, but I can't remember. Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS config issues, like matching PTR records, or setting up reasonable TTLs and SOA field values. It's right there in the data format. That's apart from where the dots and semicolons go in the database…

I'm pretty sure somewhere in my resume there's something relevant to this conversation

Hey, what a coincidence...me too!

http://www.amazon.com/Book-Webmin-Learned-Stop-Worrying/dp/1...

I've also been a contributor on an alternative DNS server in the distant past...back when it was fashionable to hate BIND.

Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS config issues, like matching PTR records, or setting up reasonable TTLs and SOA field values. It's right there in the data format.

BIND also has reasonable defaults, and there are tools (Webmin, for example, just to throw something out there completely and utterly at random) that make it easy to get all of the "reasonable default" things you've mentioned right.

You're going to have to add this to the list of topics I'm insufferable about.

OK. But you're still wrong.

Re: Sysadmin mistakes start-ups make

#78

Post by tptacek »

SwellJoe wrote: post #77
tptacek wrote: post #76

Earlier quoted context omitted.

I'm pretty sure somewhere in my resume there's something relevant to this conversation, but I can't remember. Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS config issues, like matching PTR records, or setting up reasonable TTLs and SOA field values. It's right there in the data format. That's apart from where the dots and semicolons go in the database…

I'm pretty sure somewhere in my resume there's something relevant to this conversation Hey, what a coincidence...me too! http://www.amazon.com/Book-Webmin-Learned-Stop-Worrying/dp/1... I've also been a contributor on an alternative DNS server in the distant past...back when it was fashionable to hate BIND. Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS…

Ok... but I didn't just say I was right, I offered a reason why. Why am I wrong about BIND? I'm not as up on BIND 9 as I am with 8 and 4, but last I remembered, BIND didn't automatically match PTR's to A's (one of your examples).

Re: Sysadmin mistakes start-ups make

#79

Post by tesseract »

jbert wrote: post #40
DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

I've thought this for a while. Ideally, you want to have drives of different ages, so they're at different pts of the 'bathtub curve'. If you're mirroring, you should only ever mirror a "fresh, unproven" disk with an "old stalwart" disk. Doing that also means that when your "old stalwart" becomes senile, it's paired with a younger disk. But doing that does mean rotating mirror sets when you buy a new tranch of disks.…

(In theory, as I don't run a storage farm) RAID6 might help here since if the disk rotation triggered a failure you'd have a second parity disk to fall back on. However ideally all the disks in an array would be from different lots.

Or, I believe that on some RAID controllers it is possible to add a third disk to a RAID1 pair, which means you could build the new disk before removing either of the old ones, thus there is never a single point of failure even during the disk replacement operation.

Re: Sysadmin mistakes start-ups make

#80

Post by skorgu »

DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

Legend has it that one of the early ESS systems ran into something like this.

Telephone switching systems have functionally zero downtime. They're designed to be fully modular, entirely hot swappable, the kind of thing Erlang was built for and give Z-series a run for their money. So you have this hulking room-sized[1] brute of a telephone switch which cannot ever fail and if it does so must always do so gracefully and with plenty of warning.

And one day it just falls over. No warning, no graceful failover to a redundant system, just poof gone. After much wailing and gnashing of teeth the root cause is identified: n drives were capable of failing without issue, n+1 failed simultaneously.

At this point, stories differ: this was either the beginning of a "no two drives from the same manufacturer" policy or the end of the career of a PHB who vetoed said policy on grounds of excessive cost.

[1] http://www.montagar.com/~patj/phone-switches.htm

Post reply on HN