Live data from Hacker News

Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

highscalability.com

11–20 of 33 posts

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#11
post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

Yeap, the problem was the VM. And we are going to opensource most of it.

We tried to find something that was fast, cheap to scale (in terms of servers) and easy to extend but we couldn't find any other solutions. The problem is having everything in memory and doing correlations really fast.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#13
post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

Probably has something to do with this statement, repeated many times on the page:

  We are able to do this, because starting our LISP-VM and doing all these processes on every
  request, is still many times faster that having a VM (with a garbage collector) online all the time.
Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing a full-blown VM GC system, then things become even easier. My guess is that most of the LISP code doing the data analysis is composed of calls into primitives written in C that do all the heavy lifting so the performance hit of an interpreted or bytecode-interpreted language is minimal.

Having to maintain a small LISP interpreter in C is definitely an extra thing to keep up with, but you have to balance that with the work that would normally go into avoiding long GC pauses once you start tracking a nontrivial amount of objects/data in a VM with garbage collection. Java, for instance, doesn't support separate heaps, which means that a background piggy processing thread that is haphazardly allocating objects can cause "core" threads doing I/O to pause for several seconds for full GCs. Even with Java's fairly sophisticated heap management schemes it is still very difficult to design a system to completely avoid full GC pauses. Erlang is somewhat better in this respect but introduces its own sets of problems to the mix.

In any case, I agree with your sentiment (probably would not be the angle I would have taken), but I can kind of see why the authors might have decided to go this route.

edit: formatting fixes

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#14
post #8

It would be nice to know how many man-hours it took them to get this system from 0/idea to production and how many people were in the team...

Two people, prototype in 2 months (CTO of BugSense here)

Is there a specific process you follow? Lean/agile whatever buzzword that is closest to describe how you go from idea to delivery.

How close do you work with your clients and how much is your system changing in response to feedback?

What drawbacks (if any) the current tech stack choices had so far on your?

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#15
post #5

Earlier quoted context omitted.

On linux echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout

The question was not "how do you do it once you decide" but "how do you decide on such an arbitrary replacement". Also: "Do not be confused by the /proc/sys/net/ipv4/tcp_fin_timeout config item. The FIN TIMEOUT is not the same as the TIMEWAIT length." -- http://www.stolk.org/debian/timewait.html

The value was picked heuristically after measuring through Ganglia and a set of CLI tools the average TCP dialog duration between a set of mobile devices and our database and then performing a set of tests with varying TIMEWAIT lengths. The 20 second period is a sweet (and round :)) spot with - Panagiotis Papadomitsos, Head of Infrastructure @ BugSense

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#16
post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Also, creating a HIVE/SQLish language on top of our LISP was super easy.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#17

Earlier quoted context omitted.

Two people, prototype in 2 months (CTO of BugSense here)

Is there a specific process you follow? Lean/agile whatever buzzword that is closest to describe how you go from idea to delivery. How close do you work with your clients and how much is your system changing in response to feedback? What drawbacks (if any) the current tech stack choices had so far on your?

Having a small (but very strong) R&D team helps you eliminate most of these "processes". The buzzword here is "iterations". Lot's of them.

Our clients were involved since day 0 (which I think is the key of our success btw). The first prototype (built in two days) was in Erlang using Mnesia.

The drawbacks are: + not having lots of engineers knowing C and Erlang + Outsiders having problem with LISP (that's why we implemented PIG)

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#18
post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Since garbage collecting seems such a great issue, does anyone knows of an effort to have an objective-C-like language with automatic reference counting and memory retain/release (like with clang) on the server side ?

PS : i'm speaking objective-c here because it's the only language i know that does it that way, not because of its features as a language.

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#19

Earlier quoted context omitted.

Probably has something to do with this statement, repeated many times on the page: We are able to do this, because starting our LISP-VM and doing all these processes on every request, is still many times faster that having a VM (with a garbage collector) online all the time. Self-made LISPs usually aren't that complicated to write, and if you're taking shortcuts like preallocating chunks of memory rather than writing…

Also, creating a HIVE/SQLish language on top of our LISP was super easy.

That's for the querying part, but how did you deal with indexing ? If if understand right, you seem to precompute pretty much everything you need on the fly, so I guess that means custom data structure not relying on SQL algebra, so no pkey / fkey table like and index that would let you write new queries after you've stored the data ?

Re: Using Erlang, C And Lisp To Fight The Tsunami Of Mobile Data

#20
post #7

Why using a self-made lisp here ? Why not using an existing one ? Is it because regular lisps use a VM ? But then what's the tradeoff ? On a side note, i really don't know if the savings will be worth it once the company starts growing and hiring. They'll spend a lot of time maintaining very custom code, running on a custom implementation of a language, alongside erlang codebase... Unless they open source everything…

its not just a custom lisp (which i wouldnt be too worried about) - its a custom db.

i wonder how much engineering talent is going to get sunk into writing a db and whether the management is going to get fidgety while the company's best talent is writing a db rather than the product? (especially with .. two guys)

Post reply on HN