pg, I'd love to read a post about how you've dealt with writing HN to work in one process, in 2GB of RAM, it sounds quite novel! Did you have to make manual indexes? How do you arrange the files on the file system? How do you handle voting and concurrency/file locking?
What caused that outage
41–50 of 65 posts
Re: What caused that outage
#42Well there's only one good thing for the ultimate in scalability -- arc in the cloud.
Which I believe should be called "lightning", right?
Re: What caused that outage
#43I thought 32-bit addressing gave 4GB of addresses…is there some sort of flag that's taking one bit? Not trying to be a smartass, just curious about the discrepancy.
dynamic languages end up using more memory, because extra information has to be stored about each item (I.e. Type, tc info, etc).
eeek. that should have said GC info (stupid iPod touch keyboard)
Re: What caused that outage
#44Earlier quoted context omitted.
dynamic languages end up using more memory, because extra information has to be stored about each item (I.e. Type, tc info, etc).
Down-voters, he is also right. It's common for dynamic languages to embed typing information in pointers as an optimization. For example, CLISP uses at least 2 bits to distinguish between common types. That way fixnum numbers can be recognized and added without slow memory accesses. The result is that you get less bits for the address. Hence less addressable memory.
Not to mention that many languages written in C use unions to describe their primitive -type- [ed: object]. The result is that the minimum number of bytes for storing an integer for instance, is the minimum number of bytes that can store a value of the largest type.
For example, if you have an string type, which keeps track of it's length, then you might need 8 bytes. 4 for the pointer to the string of chars, 4 for the integer to keep count.
Here's a better example from tinyscheme:
struct cell {
unsigned int _flag;
union {
struct {
char *_svalue;
int _length;
} _string;
num _number;
port *_port;
foreign_func _ff;
struct {
struct cell *_car;
struct cell *_cdr;
} _cons;
} _object;
};
typedef struct num {
char is_fixnum;
union {
long ivalue;
double rvalue;
} value;
} num;Re: What caused that outage
#45That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.
I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.
Re: What caused that outage
#46Earlier quoted context omitted.
That's why you use intelligent caching.
For a site this small, there's absolutely no reason the entire database shouldn't be cached in RAM. 32GB of RAM costs about $800. That buys you plenty of time to not have to worry about caching, and instead gives you more time to work on interesting features. For a single-person operation (or even a few people), you have to spend your time wisely.
But of course, any sane database will cache as much in RAM as possible anyway. But instead of dying when you run out of memory, it just evicts the least recently used page.
Re: What caused that outage
#47Why was the site using a 32-bit environment to begin with? Opterons have been cheap for years now, and all new Xeons are capable of 64-bit operation.
IDK if it's relevant, but 64bit can seriously suck also. If you use a lot of pointers, you just doubled the memory you need for each pointer.
In my experience, 64bit isn't a good way to save memory.
Re: What caused that outage
#48Earlier quoted context omitted.
Down-voters, he is also right. It's common for dynamic languages to embed typing information in pointers as an optimization. For example, CLISP uses at least 2 bits to distinguish between common types. That way fixnum numbers can be recognized and added without slow memory accesses. The result is that you get less bits for the address. Hence less addressable memory.
Actually, no. These tag bits are usually stored in the lowest bits which are zero for all pointers (you would be mad not to align your data structurs to the four or eight byte boundaries your hardware uses for memory access). So you get the full width for pointers, but reduced width for your fixnums, because you have to set one of the least significant bits of the machine word to one to distinguish it from a pointer.…
While that's true for some things, that doesn't account for the memory overhead of a copying garbage collector.
Re: What caused that outage
#49Earlier quoted context omitted.
The OS sometimes reserves quite a chunk. For example, Windows XP really only has 2.25GB available. Putting more RAM in is pretty useless unless you switch to 64-bit.
Can't this be changed, you know, by hacking the registry or something ?!
There's a boot-time argument that will move the barrier to 3G instead of 2G.
Re: What caused that outage
#50Yeah, it's not optimal to let your processes die from running out of memory because maybe some other non-server process actually requested the last bit of memory, and then who knows what sort of state your machine is in. How about making your restarter job notice when the machine is very close to being out of memory and preemptively kill the server then?