Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data • Hacker News Board

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#31

Post by dgk42 » Tue, Nov 27, 2012, 10:04 AM UTC

taltman1 wrote: post #25

I don't understand why the data is unstructured, and thus SQL is not an option. Isn't the data coming in from calls to the company's own API?

It has to do with semantics and the way you view your data. SQL is not a Turing-complete language (without abusing it!) and represents a subset of Relational Algebra. For us, relations might seem to resemble these of an RDBMS at first sight, but we work with them in a different way.

Also, the data format for input is not tightly-coupled with the LDB system. One can have different processing scripts (lisp) that may organize the data in a different way. Or you can write a C module that extends the underlying db engine to work with your implementation or even your database infrastructure (whether it includes SQL, NoSQL, etc).

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#32

Post by dgk42 » Tue, Nov 27, 2012, 10:31 AM UTC

bsaul wrote: post #18

stinkypete wrote: post #13

Earlier quoted context omitted.

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Since garbage collecting seems such a great issue, does anyone knows of an effort to have an objective-C-like language with automatic reference counting and memory retain/release (like with clang) on the server side ? PS : i'm speaking objective-c here because it's the only language i know that does it that way, not because of its features as a language.

We employ a technique similar to that of Obj-C in our memory handler implementation. You attach objects not to a parent object, but to a generation. We have 4 a-priori-specified generations per node (2 of them are stubborn and die during the de-initialization/clean-up phase of the node as a whole).

And this is another area where a functional paradigm makes sense. Mutations are "hidden" (and the C layer that's responsible for doing them concurrently lives in another "realm" (monad)). So, memory handling is easy and monitoring has shown us that fragmentation is kept to a minimum.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#33

Post by andor » Tue, Nov 27, 2012, 10:57 AM UTC

dgk42 wrote: post #31

taltman1 wrote: post #25

I don't understand why the data is unstructured, and thus SQL is not an option. Isn't the data coming in from calls to the company's own API?

It has to do with semantics and the way you view your data. SQL is not a Turing-complete language (without abusing it!) and represents a subset of Relational Algebra. For us, relations might seem to resemble these of an RDBMS at first sight, but we work with them in a different way. Also, the data format for input is not tightly-coupled with the LDB system. One can have different processing scripts (lisp) that may or…

I'm not sure if I understand your answer. To quote from the article again:

During our tests, we saw that SQL databases weren’t a good fit due to the fact that our data where unstructured and we needed a lot of complex “joins” (and many indexes).

I guess that you say it's unstructured because the most important parts of bug reports are stack traces and debug messages? My naive assumption is that stack traces can be stored in a relational way, and it seems that you do:

For us, relations might seem to resemble these of an RDBMS at first sight, but we work with them in a different way.

From 'unstructured', 'complex "joins"' and 'work in a different way' I infer that you do a lot of string processing on the fly, and the real problem was about precomputing everything vs. staying flexible?