Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sysadmin mistakes start-ups make

cloudkick.com

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Sysadmin mistakes start-ups make

#31

Post by vidarh »

ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

As someone else mentioned, Sqlite does indeed use lock per database file. But of course that is not an argument for it to never be used in production, but another argument for not letting sysadmins diagnose development-side issues - there are plenty of production type scenarios where Sqlite's locking is perfectly fine (but scenarios that are write heavy enough to cause lock contention is not one of them - if you use sqlite you do need to understand the locking and what workloads it is unsuitable for; especially since some bindings will give an error rather than wait for the lock)

Re: Sysadmin mistakes start-ups make

#32

Post by SwellJoe »

lsb wrote: post #29
ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

I'm getting 10k uniques a month and SQLite is working fantastically, as the db behind one Sinatra process, behind one Thin.

I'm getting 10k uniques a month

I think I see your problem.

Re: Sysadmin mistakes start-ups make

#33

Post by idlewords »

boundlessdreamz wrote: post #8
patio11 wrote: post #4

If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently makin…

Use a queue. You should never be doing time consuming method calls inside a controller anyway.

Yes, yes, yes. Beanstalkd is an easy one to set up, for example, with good Rails integration.

Re: Sysadmin mistakes start-ups make

#35

Post by SwellJoe »

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS.

Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

Re: Sysadmin mistakes start-ups make

#36

Post by astine »

vidarh wrote: post #28
michaelbuckbee wrote: post #26

Earlier quoted context omitted.

That isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?

Possibly. IBM had the infamous bad run of DeskStar (or was it another model?) drives about 8-9 years ago... We got a batch of them - in that case it wouldn't have helped you to buy them from different distributors or otherwise tried to get a different batch as the number of problematic drives was huge. At least they were extremely good about replacing them no questions asked and we got them all replaced before we los…

DeskStar (or was it another model?)

DeskStar is right. They earned the nickname "DeathStar" because they failed so often.

Re: Sysadmin mistakes start-ups make

#37

Post by gaius »

michaelbuckbee wrote: post #30

I'd guess the real number one mistake is insufficient paranoia about backups. I know lots of companies doing TDD but that have never done a full test restore from their backups.

The easiest way to do this is to make your backups the mechanism by which you refresh your Dev/QA environment from Production. It means your Ops team are very nearly doing a DR exercise every week.

Re: Sysadmin mistakes start-ups make

#38

Post by there »

SwellJoe wrote: post #35

One of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.

dns is the cause of many seemingly unrelated problems. some services (like sshd) do reverse dns lookups on connecting ips. a misconfigured dns server somewhere (or improper delegation) along the path can make this initial connection take up to 30 seconds while waiting for dns timeouts. it may look like an extremely slow/busy server, but in reality it's just sitting there doing nothing waiting for a dns reply.

tools like dnstracer (http://www.mavetju.org/unix/dnstracer.php) and dig are very useful for diagnosing these issues, but you really need to understand the fundamentals of how dns and delegation work.

Re: Sysadmin mistakes start-ups make

#39

Post by simonw »

gaius wrote: post #37
michaelbuckbee wrote: post #30

I'd guess the real number one mistake is insufficient paranoia about backups. I know lots of companies doing TDD but that have never done a full test restore from their backups.

The easiest way to do this is to make your backups the mechanism by which you refresh your Dev/QA environment from Production. It means your Ops team are very nearly doing a DR exercise every week.

I'd never heard that advice before - sounds like a great idea.

Re: Sysadmin mistakes start-ups make

#40

Post by jbert »

DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

I've thought this for a while. Ideally, you want to have drives of different ages, so they're at different pts of the 'bathtub curve'.

If you're mirroring, you should only ever mirror a "fresh, unproven" disk with an "old stalwart" disk. Doing that also means that when your "old stalwart" becomes senile, it's paired with a younger disk.

But doing that does mean rotating mirror sets when you buy a new tranch of disks. Which does put load on, which can trigger failure.

Anyone have a good plan for doing this?

Post reply on HN