Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

highscalability.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#21

Post by chongli »

stinkypete wrote: post #13
bsaul wrote: post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Greenspun's tenth rule in action!

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#22

Post by imperialWicket »

jonromero wrote: post #17
mulander wrote: post #14

Earlier quoted context omitted.

Is there a specific process you follow? Lean/agile whatever buzzword that is closest to describe how you go from idea to delivery. How close do you work with your clients and how much is your system changing in response to feedback? What drawbacks (if any) the current tech stack choices had so far on your?

Having a small (but very strong) R&D team helps you eliminate most of these "processes". The buzzword here is "iterations". Lot's of them. Our clients were involved since day 0 (which I think is the key of our success btw). The first prototype (built in two days) was in Erlang using Mnesia. The drawbacks are: + not having lots of engineers knowing C and Erlang + Outsiders having problem with LISP (that's why we imple…

This is a great write-up. Off-topic, but I'm wondering what bottlenecks you encountered with the straight Erlang + Mnesia prototype?

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#23

Post by lucian1900 »

bsaul wrote: post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

S-expressions are a nice representation for ASTs in general. At work we needed a query language for an API, so we used the data structures we had (JSON-ish) to express a little Lisp and then wrote a parser from an SQL-ish language to this Lisp.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#24

Post by jonromero »

imperialWicket wrote: post #22
jonromero wrote: post #17

Earlier quoted context omitted.

Having a small (but very strong) R&D team helps you eliminate most of these "processes". The buzzword here is "iterations". Lot's of them. Our clients were involved since day 0 (which I think is the key of our success btw). The first prototype (built in two days) was in Erlang using Mnesia. The drawbacks are: + not having lots of engineers knowing C and Erlang + Outsiders having problem with LISP (that's why we imple…

This is a great write-up. Off-topic, but I'm wondering what bottlenecks you encountered with the straight Erlang + Mnesia prototype?

Mnesia is by far the fastest (and easiest to use) thing we've ever tried. The only problem was that it was more expensive to keep data in memory and that pointer arithmetic is so damn fast. And (the most important thing) is the 2GB limit Dets has :/

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#26

Post by krichman »

bsaul wrote: post #18
stinkypete wrote: post #13

Earlier quoted context omitted.

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Since garbage collecting seems such a great issue, does anyone knows of an effort to have an objective-C-like language with automatic reference counting and memory retain/release (like with clang) on the server side ? PS : i'm speaking objective-c here because it's the only language i know that does it that way, not because of its features as a language.

I'm pretty sure CPython reference counts objects. Also Objective-C -- as in GNUStep, if you aren't using a Mac server.

EDIT: There's a downside to reference counting in that either you or the runtime does the retain/release calls very often, whereas a mark-and-sweep collector would just touch each object once per collection. So reference counting might not be faster overall, but typically avoids long pauses. What you probably really want is something like Azul's JVM, which has a kernel extension so they can collect memory concurrently, resulting in shorter pauses than sweeping and faster overall time than reference counting.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#27

Post by krichman »

priestjim wrote: post #15
saurik wrote: post #5

Earlier quoted context omitted.

The question was not "how do you do it once you decide" but "how do you decide on such an arbitrary replacement". Also: "Do not be confused by the /proc/sys/net/ipv4/tcp_fin_timeout config item. The FIN TIMEOUT is not the same as the TIMEWAIT length." -- http://www.stolk.org/debian/timewait.html

The value was picked heuristically after measuring through Ganglia and a set of CLI tools the average TCP dialog duration between a set of mobile devices and our database and then performing a set of tests with varying TIMEWAIT lengths. The 20 second period is a sweet (and round :)) spot with - Panagiotis Papadomitsos, Head of Infrastructure @ BugSense

I was hoping it got chosen empirically. It's awesome that they were wise enough to not pick some theoretical best and calling it a day.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#30

Post by dgk42 »

zhoudh wrote: post #29

I am interest in the missing part: what data structures/layouts are used inside LDB? Cause for In-Memory DB, CPU cache miss plays a key role in executing query. And how much data are touched to execute a single query in common case?

Cache misses are not compared to initializing the lisp engine. So, we don't worry so much about them. Also, since the data organization is up to us, we touch only what is necessary (well, almost! We are talking about fine-grained locking here, but we modify (mutate) only the datum that needs to be changed).
Post reply on HN