Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sysadmin mistakes start-ups make

cloudkick.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Sysadmin mistakes start-ups make

#41

Post by wmf »

michaelbuckbee wrote: post #26
DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

That isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?

Would it make sense to use drives from more than one company as they would very different failure characteristics?

This is probably difficult to take advantage of unless you setup your RAID layout very carefully.

Re: Sysadmin mistakes start-ups make

#42

Post by simonw »

swombat wrote: post #17
patio11 wrote: post #15

Earlier quoted context omitted.

You just described my exact setup. However, my understanding is that Delayed::Job's worker threads have a full Rails environment in them, and if this blog post is correct and I am indeed forking that entire Rails process for every call out to ImageMagick, my vague recollections of what a fork entails suggest to me that the Ghosts of C Programmers Past are going to visit a terrible vengeance upon me.

Only if you run out of memory. But with DJ at least you should be forking only one call at a time, rather than multiple, like you might from the controller itself. So although you'll end up using more memory, it'll only be one extra rails process, not 4. May not still be ideal... interested to hear other people's ideas.

Use http://github.com/documentcloud/cloud-crowd

Re: Sysadmin mistakes start-ups make

#43

Post by SwellJoe »

there wrote: post #38
SwellJoe wrote: post #35

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

dns is the cause of many seemingly unrelated problems. some services (like sshd) do reverse dns lookups on connecting ips. a misconfigured dns server somewhere (or improper delegation) along the path can make this initial connection take up to 30 seconds while waiting for dns timeouts. it may look like an extremely slow/busy server, but in reality it's just sitting there doing nothing waiting for a dns reply. tools l…

Yes, I should have been emphatic that part of the problem with not understanding DNS is that if you don't understand it, you might not even realize that your problem is DNS-related.

Whenever I hear, "X is slow!", my first response tends to be, "Are you sure it's not DNS instead of X?" About 50% of the time, DNS misconfiguration is at least a component of the problem if not the entirety of it.

DNS touches every service on the Internet. If you get it wrong, you break every service, sometimes in subtle ways.

Re: Sysadmin mistakes start-ups make

#44

Post by blasdel »

michaelbuckbee wrote: post #26
DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

That isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?

It's a terrible idea to mix and match drive models, because they'll have drastically different performance characteristics, and if you're lucky you'll only get a little worse than the lowest common denominator on all axes.

What quality OEMs do is make sure they never ship you drives from the same manufacturing batch in one enclosure.

Re: Sysadmin mistakes start-ups make

#46

Post by silentbicycle »

pquerna wrote: post #24
ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

SQLite locking: http://www.sqlite.org/lockingv3.html """ An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held. """ But compared to something like MySQL w/ InnoDB (or postgr…

Like many of SQLite's other quirks, this is because SQLite is designed to accommodate embedded usage.

Re: Sysadmin mistakes start-ups make

#47

Post by julio_the_squid »

Yep, #1 happened to me the other day. We hit our Apache server limit of 256 and the site slowed to a crawl. I'm not really sure what was causing the load to be like 50-90, but requests were quite delayed waiting for an open process (keepalive was at 5 secs).

Indeed, my first idea was indeed to install nginx for images really quick. However, I have no experience with nginx. Thankfully, we had a spare server and I offloaded the images to there for now... Throwing more hardware at the problem usually works.

Re: Sysadmin mistakes start-ups make

#48

Post by Periodic »

peterwwillis wrote: post #9

The memory use is not accurate unless you take shared pages into account. Copy-on-write will make it look like each apache child is using 40MB, when really it's only 10MB private RSS. Use a RSS-calculating script ( http://psydev.syw4e.info/new/misc/meminfo.pl ) to determine the close-to-real memory use. If you don't calculate your maximum memory use correctly you will run into swap with traffic peaks. Also keep in mi…

This stood out to me as well. I like the script to actually calculate real usage. Modern operating systems are smarter than I am when it comes to memory management.

What you don't want is to have anything you use more than once a minute in swap, and preferably only the stuff you don't plan on using for an hour (i.e. not any time soon). That probably means you want your main application and web server in memory all the time. If there are pieces of it that are unused and you're hitting a resource cap then you have something mis-configured.

RAM is also dirt cheap right now, making it often easier to add RAM than to optimize slightly sloppy code.

Re: Sysadmin mistakes start-ups make

#49

Post by jrockway »

Fork is actually a very fast system call. It never blocks, and (on Linux), only involves copying a very small amount of bookkeeping information. If you exec right after the fork, there is basically no overhead.

However, forking a new shell to parse "mv foo bar" is more expensive than just using the rename system call. And it's easier to check for errors, and so on.

SQLite is also not as slow as people think it is; you can easily handle 10s of millions of requests per day with it. If your application's semantics require table locks, MySQL and Postgres are not going to magically eliminate competition for locks. It's just that they both pick very weak locking levels by default. (They run fast, but make it easy to corrupt your data. Incidentally, I think they do this not for speed, but so that transactions never abort. Apparently that scares people, even though it's the whole point of transactions. .)

Most of my production apps are SQLite or BerekelyDB, and they perform great. I am not Google, however.

Re: Sysadmin mistakes start-ups make

#50

Post by Huppie »

SwellJoe wrote: post #35

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

I'm not sure which book you are talking about but my guess would be "DNS and Bind". http://oreilly.com/catalog/9780596100575/
Post reply on HN