What caused that outage • Hacker News Board

Re: What caused that outage

#61

Post by jrockway » Fri, Mar 13, 2009, 1:32 AM UTC

lacker wrote: post #60

pg wrote: post #58

Earlier quoted context omitted.

I would be keeping everything in memory (using lazy loading) regardless. It makes life simpler.

You wouldn't use SQL? Something like django or rails is pretty handy if you are already using those languages, as annoying as it can be.

SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model.

Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.

Re: What caused that outage

#62

Post by elibarzilay » Fri, Mar 13, 2009, 1:51 AM UTC

rw wrote: post #21

Jebdm wrote: post #20

Earlier quoted context omitted.

If I'm not mistaken, "to battletest a new language" was the purpose of building this.

Keep in mind that mzscheme is battle tested. Arc is "just" a bunch of macros on top of PLT Scheme; what gives HN its [under]performing character is that foundational JIT compiler and 3m GC.

In general, 3m is performing very well, much better than the conservative collector plt used to use by default (the boehm gc). As for the jit: many of the benefit of the jit are irrelevant in Arc since it doesn't use mzscheme modules; this is in addition to using an old mzscheme version, when the jit was rather new -- many many improvements were done since then.

Re: What caused that outage

#63

Post by elibarzilay » Fri, Mar 13, 2009, 1:59 AM UTC

Something that could be done for now, is to write a piece of mzscheme code that "marshalls" the data in (utf-8-encoded) byte-strings. Assuming that most of the 2gb is made of strings, and that these strings are mostly ascii, this should reduce the consumption by close to a factor of 4.

(I can imagine an interface that is transparent at the arc level, where are strings are just passed to the backend and retrieved from it, and the backend converts them to and from byte strings. Later on it could change to use a FS or a DB or whatever.)

Re: What caused that outage

#64

Post by lacker » Fri, Mar 13, 2009, 3:24 AM UTC

jrockway wrote: post #61

lacker wrote: post #60

Earlier quoted context omitted.

You wouldn't use SQL? Something like django or rails is pretty handy if you are already using those languages, as annoying as it can be.

SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model. Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone t…

Actually, most of the websites I write don't use SQL.

I don't think your "poor fit" comment is true. You don't need to model the threaded nature of comments at all in your database. Since there's a relatively small number of comments per post, and most access to comments is probably show-me-all-comments-for-this-submission, you usually just need a single foreign key, each comment to its original grandparent submission.

In this specific context the relevant advantage of SQL over rolling your own is that there is less distinction between keeping data in memory and on disk. Otherwise, you can certainly do anything with memory + SQL that you can with memory + disk.

Re: What caused that outage

#65

Post by jrockway » Sat, Mar 14, 2009, 4:00 AM UTC

lacker wrote: post #64

jrockway wrote: post #61

Earlier quoted context omitted.

SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model. Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone t…

Actually, most of the websites I write don't use SQL. I don't think your "poor fit" comment is true. You don't need to model the threaded nature of comments at all in your database. Since there's a relatively small number of comments per post, and most access to comments is probably show-me-all-comments-for-this-submission, you usually just need a single foreign key, each comment to its original grandparent submissio…

Sure, you can store it in a relational database, but you have to map your data to some structure that it doesn't have. That's what we call a hack.

If you use an object database, you can store your data exactly as it is represented in memory. That is much cleaner, IMO.

(Relational databases are like programming languages. Just like you can use any programming language for any task, and you can hack any data you want into the relational model. But that's not always the best way. Sometimes a key/value store, or an object store, or a document store is a better model. Using a better model means you need to write less code, which means your app will have fewer bugs.)