NIGGERS!!!!!!!!!!!!!!!!!!!!!!!
Sysadmin mistakes start-ups make
81–90 of 95 posts
Re: Sysadmin mistakes start-ups make
#82Earlier quoted context omitted.
I'm pretty sure somewhere in my resume there's something relevant to this conversation Hey, what a coincidence...me too! http://www.amazon.com/Book-Webmin-Learned-Stop-Worrying/dp/1... I've also been a contributor on an alternative DNS server in the distant past...back when it was fashionable to hate BIND. Meanwhile, the reason I brought up djbdns is that it has automatic best-practice behavior for a lot of basic DNS…
Ok... but I didn't just say I was right, I offered a reason why. Why am I wrong about BIND? I'm not as up on BIND 9 as I am with 8 and 4, but last I remembered, BIND didn't automatically match PTR's to A's (one of your examples).
Because I'm not talking about BIND at all. I'm talking about DNS. I don't care what DNS server people are using (we also support PowerDNS, and we also get questions about djbdns, even though we don't support it). These are the mistakes people make, regardless...and it's due to fundamental misunderstandings of how DNS does what it does, and rarely has anything to do with syntax (most of the people I interact with are using Webmin, which hides the configuration file entirely, and so the semi-colons and dots are irrelevant because Webmin always gets them right; and Webmin can also automatically manage PTRs).
Re: Sysadmin mistakes start-ups make
#83Earlier quoted context omitted.
Ok... but I didn't just say I was right, I offered a reason why. Why am I wrong about BIND? I'm not as up on BIND 9 as I am with 8 and 4, but last I remembered, BIND didn't automatically match PTR's to A's (one of your examples).
Because I'm not talking about BIND at all. I'm talking about DNS. I don't care what DNS server people are using (we also support PowerDNS, and we also get questions about djbdns, even though we don't support it). These are the mistakes people make, regardless...and it's due to fundamental misunderstandings of how DNS does what it does, and rarely has anything to do with syntax (most of the people I interact with are…
Ok, but I'm specifically not talking about the semicolons and dots. I'm talking about the semantic issues you brought up, like matching PTR's to A's, or the fact that "You may omit ttl; tinydns-data will use default cache times, carefully selected to work well in normal situations.", or the fact that you can't forget to bump serial numbers in djbdns files.
I'm not needling you, Joe. I just genuinely don't know what --- apart from picking up the delegation for your domain name --- the common DNS errors are that djbdns doesn't address. I'm not saying there aren't any; I'm just asking you to say what they are.
Also, shouldn't you just go ahead and support djbdns? It always seemed to me like Bernstein went way out of his way to make it easy to drive tinydns from other programs. Do you really get more requests for PowerDNS than djbdns?
Re: Sysadmin mistakes start-ups make
#84If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently makin…
The solution to use an image processing library such as RMagick, http://rmagick.rubyforge.org/
Last I tried RMagick, it leaked significantly. Definitely not something I want to use in a long-lived process. I remember having to fork to use it, to work around the memory leak. If you don't need the fancier operations, there are lighter image manipulation gems out there that do just the basics, but without leaking. e.g. ImageScience http://seattlerb.rubyforge.org/ImageScience.html
Re: Sysadmin mistakes start-ups make
#85Earlier quoted context omitted.
Because I'm not talking about BIND at all. I'm talking about DNS. I don't care what DNS server people are using (we also support PowerDNS, and we also get questions about djbdns, even though we don't support it). These are the mistakes people make, regardless...and it's due to fundamental misunderstandings of how DNS does what it does, and rarely has anything to do with syntax (most of the people I interact with are…
Ok, but I'm specifically not talking about the semicolons and dots. I'm talking about the semantic issues you brought up, like matching PTR's to A's, or the fact that "You may omit ttl; tinydns-data will use default cache times, carefully selected to work well in normal situations.", or the fact that you can't forget to bump serial numbers in djbdns files. I'm not needling you, Joe. I just genuinely don't know what -…
I think what SwellJoe is getting at is that different software won't address misunderstandings and assumptions. Eg: There's a lot of folks out there who aren't aware that they should incrementally drop a records TTL prior to changing it's data so as to control the amount of time the record is in flux. See http://search.twitter.com/search?q=dns+waiting+OR+propagate or similar.
Re: Sysadmin mistakes start-ups make
#86Earlier quoted context omitted.
Because I'm not talking about BIND at all. I'm talking about DNS. I don't care what DNS server people are using (we also support PowerDNS, and we also get questions about djbdns, even though we don't support it). These are the mistakes people make, regardless...and it's due to fundamental misunderstandings of how DNS does what it does, and rarely has anything to do with syntax (most of the people I interact with are…
Ok, but I'm specifically not talking about the semicolons and dots. I'm talking about the semantic issues you brought up, like matching PTR's to A's, or the fact that "You may omit ttl; tinydns-data will use default cache times, carefully selected to work well in normal situations.", or the fact that you can't forget to bump serial numbers in djbdns files. I'm not needling you, Joe. I just genuinely don't know what -…
OK, let's get specific:
PTR requires delegation from the authoritative server for the IP address, and this is wholly separate from the registrar process. Many folks don't understand how PTR works vs. how standard A records work. If you don't understand it, you don't know you need to talk to your hosting provider about this delegation.
Many folks also don't understand how the PTR is used in validating sending mail servers. And many folks have a hard time grasping that you only configure one PTR for each IP address (see? I'm talking fundamental misunderstandings of DNS here, and no DNS server or GUI can fix them, though gods know we've tried). Finally, you'd be surprised how many people intentionally configure a PTR that does not resolve in the other direction, thus breaking mail.
Most folks are dealing with many domains on a single IP address. This further confuses folks with regard to the PTR. Which name for the IP? Many folks are baffled. A basic understanding of DNS would resolve this (no pun intended). As I mentioned above, with an understanding of the way it works, most DNS problems are completely transparent.
Default TTL doesn't solve a user not understanding that when they change IP addresses, it can take as much as days for the rest of the world to know about those changes (first world will know within hours; third world caches much longer and occasionally ignores TTL). Again, I'm talking about users fundamentally not understanding how DNS works and that it is a heavily cached protocol. If you don't understand those things, having a default TTL doesn't help you when it's time to migrate to a new data center. Webmin also provides default ttl values.
Serial numbers. I have never once mentioned serial numbers in this discussion. I can't remember the last time I've heard someone who had a problem with serial numbers...actually, now that I'm thinking of it, I do remember. A user (I think a recovering djbdns user, actually) who thought they knew more than they really did wanted to argue about RFCs and whether an incrementing number or a datestamp was valid (I don't remember which, but the form he was arguing was invalid was also provably valid, based on examples found within the RFC). But, this is the kind of silly stuff that I'm not talking about. Syntax is irrelevant to my suggestion that people read DNS and BIND. Oh, and Webmin handles serial numbers in BIND, in just about any format that is valid according to the RFC.
Some others issues that I see people make:
Incorrect NS, MX, etc. records. djbdns doesn't babysit the records to make sure they point to working servers. Users occasionally try to use IPs directly in these records. BIND will error on this, as expected, but it still confuses people who don't grasp basic concepts.
Performance. People who don't understand DNS often think they can get out of it by relying on someone else for their DNS service. Like the overworked DNS servers provided free with their registrar or hosting account. On top of this, these are also the same people that make all the dumb mistakes mentioned above, compounding their confusion into a perfect storm of "nothing was working, so I reinstalled my operating system and now nothing works!" stupidity.
Fundamental misunderstandings of the protocol and the way DNS does what it does are what I'm talking about. Some DNS servers are easier to configure than others...I'll concede that. Doesn't matter. When you set out on a journey, you kinda have to have a vague notion of where you're going, or you probably aren't going to get there, even if you have a GPS.
Also, shouldn't you just go ahead and support djbdns?
Why? All the security issues that plagued BIND 4, and to some degree BIND 8, have mostly been long resolved, and BIND 9 is, by far, the world's most popular DNS server. djbdns these days is obscure, at best. You and I remember the time when it was relevant...but I'm involved enough in this aspect of the industry to know that the ship has long since sailed for djbdns; new users are not adopting djbdns on any scale worth talking about, though plenty of cantankerous old-timers keep hanging on. We get maybe one person every six months to ask about djbdns. There are so many things ahead of djbdns in our queue that it will likely never make it to the head.
nginx is our next major endeavor (which gets requested every couple of days, and has real advantages over Apache for some of our real paying customers). nginx is actually a much bigger job than djbdns would be, since DNS is already fully abstracted out of Virtualmin because of the PowerDNS module, while web service with Apache is pretty deeply ingrained. But I can pretty clearly see an upside for us in supporting it that I do not see with djbdns.
Also, the historic license stupidity of djb software has guaranteed that all of it would fall into obscurity. I guess most of it is public domain now, but I just don't see much new interest in their use.
Do you really get more requests for PowerDNS than djbdns?
Yes, we did, but we also got money for PowerDNS support. It wouldn't have happened without it being fully sponsored by a hosting provider that wanted to use PowerDNS. We don't do contract work any more, but we could help you find a developer to add djbdns support, if you'd like to see it in Webmin and Virtualmin. Or, if you know Perl, we'd be happy to assist you with getting up to speed on the module API.
Actually, it looks like there is already a third party Webmin module, of some sort, for djbdns. Though it looks like it hasn't been updated in years, and doesn't have a lot of discussion on the web about it.
Just so you don't think we hate djb, we do fully support qmail, though I happen to prefer Postfix by a large margin (and I doubt we would spend significant time on qmail today...the userbase is a fraction of what it used to be, and we rarely get questions about it).
Anyway, I'm still not really interested in talking about the relative merits of DNS servers. I don't care. They all work acceptably well, at this point, and I rarely see a BIND configuration file (I don't have a problem with BIND configuration files, but why bother? I've got tools for that).
I just wanted people to understand the most fundamental building block of the Internet a little better, and I pointed to the best book on the subject. Complaining about DNS and BIND because it uses BIND configuration files for the examples is like complaining about TAoCP because it uses MIX rather than Python (or whatever) for the examples. There is no better book on the subject of DNS, or if there is, I have never seen it.
Re: Sysadmin mistakes start-ups make
#87Fork is actually a very fast system call. It never blocks, and (on Linux), only involves copying a very small amount of bookkeeping information. If you exec right after the fork, there is basically no overhead. However, forking a new shell to parse "mv foo bar" is more expensive than just using the rename system call. And it's easier to check for errors, and so on. SQLite is also not as slow as people think it is; yo…
> SQLite is also not as slow as people think it is; you can easily handle 10s of millions of requests per day with it Scaling relational databases to the point of 10s of millions of requests is extremely non-trivial. Unless you can show me personally or can show me evidence otherwise, don't make this claim. You're doing a disservice to the people that have worked countless hours to eke every last millisecond of perfo…
Scaling relational databases to the point of 10s of millions of requests is extremely non-trivial.
Simply not true. That level of database activity is bread-and-butter for many people these days. Hundreds of millions, non-trivial in the sense that it will cost you a pile of money, but these days, basically straightforward, especially if you've done it before. Thousands of millions, now that's where things get interesting.
Oh, and he's talking about requests. The trading systems I've worked on can and do handle hundreds of millions of commits per day.
Re: Sysadmin mistakes start-ups make
#88Is this a example of a knowledge level of modern sysadmin? If so, we're in trouble. =)
Sysadmin should be able to think in terms of data flows, which means memory management, data partitioning, and network stack usage, able to put different types of data into different kinds of storage, and understand the role of cache and how data should be access.
Packages are just a tools.
Re: Sysadmin mistakes start-ups make
#89Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.
You frequently see RAID cascade failures.
Drive A fails. You pop it out, put in a new disk. Machine starts rebuilding RAID array. This involves a ton of reads from other disks. Under the increased strain, Drive B, which was already on the edge, also fails.
And so on.
I'm no longer a believer in redundant disks. Redundant machines would be better.
Re: Sysadmin mistakes start-ups make
#90FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…
I read an interesting performance comparison a while back (which I can't find now) which had some surprising results for sqlite with concurrent access. As I recall, it turned out that it was much faster to close the db connection and open it again for each operation than to keep a connection open and rely on the file locking to mediate access.