Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sysadmin mistakes start-ups make

cloudkick.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Sysadmin mistakes start-ups make

#21

Post by ajross »

FTA:

However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock

I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or across a network. Maybe the author is thinking fcntl() locking is somehow the only option.

I guess the corrolary to this article has to be "Don't let your startup's sysadmins diagnose development-side issues."

Re: Sysadmin mistakes start-ups make

#22

Post by polvi »

tptacek wrote: post #13
polvi wrote: post #7

Earlier quoted context omitted.

The solution to use an image processing library such as RMagick, http://rmagick.rubyforge.org/

Calling into RMagick/ImageMagick from inside the request/response cycle is probably even worse than shelling out, because ImageMagick does grievous damage to your runtime.

I guess it all depends how you design it and what you are doing. I would have to agree with others, the out of request cycle image processing solutions are definitely the right way to go overall.

Re: Sysadmin mistakes start-ups make

#23

Post by jeremyw »

patio11 wrote: post #15
swombat wrote: post #10

Earlier quoted context omitted.

I'd use something like DelayedJob and send_later the call to your image processing stuff, that way the forking happens out of the request path, at least.

You just described my exact setup. However, my understanding is that Delayed::Job's worker threads have a full Rails environment in them, and if this blog post is correct and I am indeed forking that entire Rails process for every call out to ImageMagick, my vague recollections of what a fork entails suggest to me that the Ghosts of C Programmers Past are going to visit a terrible vengeance upon me.

The fork+exec is efficient. The blog post compares things without units. Forks (principally page table copies w/copy-on-write in effect) are measured in microseconds and the exec is your standard binary startup time. While you don't want to put a synchronous fork/exec in the way of 5,000 reqs/sec, it will be a trivial part of your asynchronous imagemagick processing.

At scale, you might care about the imagemagick startup latency, but not the forking.

Re: Sysadmin mistakes start-ups make

#24

Post by pquerna »

ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

SQLite locking: http://www.sqlite.org/lockingv3.html

""" An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held. """

But compared to something like MySQL w/ InnoDB (or postgres, or Cassandra, or BerkeleyDB), which all have something closer to Row Level or Page Level locking, SQLite's concurrency for server side applications is a serious deficiency.

Yes, there are lots of ways to have fine grained locking, SQLite just doesn't do them.

Re: Sysadmin mistakes start-ups make

#25

Post by peterwwillis »

ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

I guess the corrolary to this article has to be "Don't let your startup's sysadmins diagnose development-side issues."

You'll have to add "Make sure your developers can diagnose development-side issues" to the list as well. Most web app developers I have met do not know how to diagnose problems, or simply defer immediately to the sysadmins if there's no syntax errors or logs to refer to.

Re: Sysadmin mistakes start-ups make

#26

Post by michaelbuckbee »

DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

That isn't something I'd ever thought about until now.

Would it make sense to use drives from more than one company as they would very different failure characteristics?

Re: Sysadmin mistakes start-ups make

#27

Post by thwarted »

rythie wrote: post #19

This seems like an odd section of sysadmin mistakes - I would have thought there are some other ones being made more often.

Especially since the third one is a developer mistake that, as a sys admin and developer, I've had to point out to developers not to do -- but for security reasons, not because fork is oh-so-super expensive (even though it can be).

Also, there is no "system" system call. "system" is a library call that forks and execs a shell to evaluate and execute a string. Having a sys admin that doesn't know the difference may be the biggest sys admin mistake you could make. There are a lot of library wrappers for system calls, but these are documented in section 2 of the man pages as system calls.

Re: Sysadmin mistakes start-ups make

#28

Post by vidarh »

michaelbuckbee wrote: post #26
DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

That isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?

Possibly. IBM had the infamous bad run of DeskStar (or was it another model?) drives about 8-9 years ago... We got a batch of them - in that case it wouldn't have helped you to buy them from different distributors or otherwise tried to get a different batch as the number of problematic drives was huge. At least they were extremely good about replacing them no questions asked and we got them all replaced before we lost data.

Re: Sysadmin mistakes start-ups make

#29

Post by lsb »

ajross wrote: post #21

FTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or…

I'm getting 10k uniques a month and SQLite is working fantastically, as the db behind one Sinatra process, behind one Thin.
Post reply on HN