Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What caused that outage

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: What caused that outage

#51

Post by lacker »

jrockway wrote: post #45
mixmax wrote: post #3

That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.

I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.

Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory.

That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.

Re: What caused that outage

#52

Post by jcl »

DomesticMouse wrote: post #31
ajkirwin wrote: post #13

You aren't running this in the standard language/caching/db setup, then? You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.

Dude, frack no. Having everything in a database means you are tied to the speed of disk, instead of the speed of ram. The biggest issue with being purely ram resident is handling multi-threaded updates across your ram resident dataset. You could always use Erlang...

If everything is purely RAM resident, wouldn't you lose it all when the server gets wedged?

Re: What caused that outage

#53

Post by lacker »

catch404 wrote: post #38
paulgb wrote: post #29

Earlier quoted context omitted.

If you're really curious and have the time, you can download the source: http://arclanguage.org/install

It'd be interesting to read a write up on the concepts used (such as the in memory database) and the server restart mechanism. It'd be hard to tell from the source for non Lisp programmers.

An in-memory database is pretty simple - it's just a bunch of hash tables.

A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.

Re: What caused that outage

#54

Post by jcl »

vinutheraj wrote: post #34
protomyth wrote: post #15

Earlier quoted context omitted.

The OS sometimes reserves quite a chunk. For example, Windows XP really only has 2.25GB available. Putting more RAM in is pretty useless unless you switch to 64-bit.

Can't this be changed, you know, by hacking the registry or something ?!

No. The Windows and Linux kernels are hardwired to reserve a huge chunk of virtual addresses for kernel memory space. Windows can be toggled between reserving 2Gb and 1Gb (the latter of which Linux does by default). I assume this is so that a system memory address can be identified by testing a couple bits, and the minimum size is presumably limited by the chunk of addresses eaten up by memory-mapped devices like video cards.

Here's a page with some more details: http://news.ycombinator.com/item?id=452005

Re: What caused that outage

#55

Post by sounddust »

jcl wrote: post #52
DomesticMouse wrote: post #31

Earlier quoted context omitted.

Dude, frack no. Having everything in a database means you are tied to the speed of disk, instead of the speed of ram. The biggest issue with being purely ram resident is handling multi-threaded updates across your ram resident dataset. You could always use Erlang...

If everything is purely RAM resident, wouldn't you lose it all when the server gets wedged?

No. You journal the changes to disk. You therefore only need disks fast enough to keep up with the journal. If you get into a situation where your disks can't keep up with the journal, your site is probably big and popular enough that you can afford to hire DBAs to go from there.

Re: What caused that outage

#56

Post by lsc »

hey, so I'd be happy to donate 16GB worth of Xen instances to the project,just to say I did, if you need mirrors. my boxes are 32GB ram/8 core, and the CPU is proportional, so if you want 4x4GB instances over separate servers, I can do that. (well, you will have to wait a bit for me to put up the 4th server.)

Re: What caused that outage

#57

Post by jrockway »

lacker wrote: post #51
jrockway wrote: post #45

Earlier quoted context omitted.

I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.

Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory. That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.

I suspect if pg was not using Arc he would not be keeping everything in memory.

I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.)

With that in place, the site should perform fine.

Re: What caused that outage

#58

Post by pg »

lacker wrote: post #51
jrockway wrote: post #45

Earlier quoted context omitted.

I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.

Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory. That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.

I would be keeping everything in memory (using lazy loading) regardless. It makes life simpler.

Re: What caused that outage

#59

Post by catch404 »

lacker wrote: post #53
catch404 wrote: post #38

Earlier quoted context omitted.

It'd be interesting to read a write up on the concepts used (such as the in memory database) and the server restart mechanism. It'd be hard to tell from the source for non Lisp programmers.

An in-memory database is pretty simple - it's just a bunch of hash tables. A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.

Thanks, I understood it uses hash tables - I've done the same myself, but never on such a large scale. This post sums it up: http://news.ycombinator.com/item?id=513259

Re: What caused that outage

#60

Post by lacker »

pg wrote: post #58
lacker wrote: post #51

Earlier quoted context omitted.

Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory. That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.

I would be keeping everything in memory (using lazy loading) regardless. It makes life simpler.

You wouldn't use SQL? Something like django or rails is pretty handy if you are already using those languages, as annoying as it can be.
Post reply on HN