Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Erlang Shell

medium.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: The Erlang Shell

#51

Post by tbrownaw »

chubot wrote: post #26
rdtsc wrote: post #11

> When systems have faults due to concurrency and distribution, debuggers will not work. Those are very tricky indeed, mix in threads with pointers and a system becomes haunted. "This one customer noticed a crash, on this hardware, after running for 10 weeks, but ... we couldn't reproduce it". "Oh I suspect there is a memory overwrite or use after free that corrupted the data". People start doubting their sanity -- "…

Nitpick: mixing threads with pointers isn't really the problem; it's mixing threads with shared mutable state. A concurrency idiom I've gotten a lot of mileage out of is to use threads, without shared state, but with pointers. Instead of shared state, you use a message passing style where you pass pointers over queues. Threads only mutate data that they are explicitly handed via a queue. After putting an object on a…

Instead of shared state, you use a message passing style where you pass pointers over queues.

If you don't have a queue library and don't want to risk getting the locking wrong, you can get one from pipe(2). While simultaneous writes/reads are in theory permitted to be interleaved, the implementation would have to be actively insane to interleave anywhere other than page boundaries.

...I actually did this when I was at university, for the first class that covered locking and concurrency. Amazingly enough, I didn't fail that assignment for delegating what we were supposed to be learning to the OS.

The logical structure of such a program is not really that different than an Erlang program, I imagine. It just works a little better with existing C/C++ codebases, and is potentially faster in some cases.

It has a completely different character than spaghetti written using threads and shared state, and can be made extremely reliable.

Maybe this is why I don't get all the C++-hate? The things that make code unreliable or thread-unsafe tend to also make it hard to read and think about (even without threads), so I just find some other way that doesn't give me a headache.

Interestingly code that doesn't give me a headache tends to have a strong resemblance to functional-style code, even tho actual pure functional code also tends to be mildly headache-inducing.

Re: The Erlang Shell

#52

Post by sea6ear »

klibertp wrote: post #48
logicchains wrote: post #43

Earlier quoted context omitted.

As someone who's written a bit of Common Lisp, I can say that the macros are a pretty awesome feature to use. Common Lisp has the advantage over Clojure and Scheme that the SBCL compiler is incredibly fast; it can come close to C#/F# on Mono for some tasks, in spite of Lisp being a dynamic language. It also compiles code and expands macros much faster. It's got more power than the other lisps, but it's also less eleg…

> I can say that the macros are a pretty awesome feature to use Erlang has them in the form of parse transforms and Elixir has them too and makes them easy to use. Not as easy as sexp based languages, but almost.

There is also LFE (Lisp Flavored Erlang) [0] which gives lisp style macros and some other goodies to Erlang

[0] http://lfe.github.io/

Re: The Erlang Shell

#53

Post by phamilton »

simula67 wrote: post #30
phamilton wrote: post #17

Earlier quoted context omitted.

Is there a mechanism to migrate existing data to the new code?

If you have external state stored in say a database, I think you have to control your deployments to migrate the schema before the model objects can be deployed.

So no.

One of the beauties of Erlang's hot code loading is the ability to migrate internal application state. An example:

I'm storing a lookup table going from key to value as a tuple {key, value}. With a change going out, I'm going to start doing {key, {value, other-value}}. In Erlang there's an easy pattern in place to do that. Just provide a function that goes from {key, value} to {key, {value, sensible-default}} and as part of the code reloading process that function will be run before the new code starts executing.

Re: The Erlang Shell

#54

Post by skywhopper »

Reading about "correct software" as if it were an realistic goal makes me itch. Sure, we should be striving for as few bugs as possible, but any non-trivial application will never be made "correct", if only because it'd be impossible to write the full specification to compare it to.

Re: The Erlang Shell

#55

Post by chubot »

tbrownaw wrote: post #51
chubot wrote: post #26

Earlier quoted context omitted.

Nitpick: mixing threads with pointers isn't really the problem; it's mixing threads with shared mutable state. A concurrency idiom I've gotten a lot of mileage out of is to use threads, without shared state, but with pointers. Instead of shared state, you use a message passing style where you pass pointers over queues. Threads only mutate data that they are explicitly handed via a queue. After putting an object on a…

Instead of shared state, you use a message passing style where you pass pointers over queues. If you don't have a queue library and don't want to risk getting the locking wrong, you can get one from pipe(2) . While simultaneous writes/reads are in theory permitted to be interleaved, the implementation would have to be actively insane to interleave anywhere other than page boundaries. ...I actually did this when I was…

Yup, pipe() is actually a great, minimal solution. POSIX guarantees PIPE_BUF to be at least 512 bytes, so you are guaranteed atomicity for any size pointer. I think on Linux the unit of atomicity is 4096 bytes, which is a page.

Yeah, there are a million ways to use C++. To be fair, most people have their architecture forced upon them, rather than creating it from scratch. And the C++ world does have a culture of spaghetti with shared mutable state and threads. In Erlang you have the opposite culture. So I can understand the association that people make with a language, even though it's more a matter of culture.

In addition, C++ doesn't even have a thread safe queue in the STL (although I was pointed to a draft for one in the next version of C++ by a coworker). pipe() is a good solution I would use, but not portable to other OSes.

Re: The Erlang Shell

#56

Post by pessimizer »

phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

>In Rails, a process is short-lived by design (the lifecycle of a request).

I'm not sure if this is by design, or a limitation of Ruby.

>short-lived processes aren't the best approach for a web service.

Not for me - I'm building all of my web services in Erlang these days, and there's little to no relationship between the life of most processes in the service and the life of requests. Although each request is usually serviced by its own process, that process may interact with other processes that may have been alive for as long as the application has been running. For example, my RAM caching strategies are usually written as part of the application, not put in front of them. Makes pseudo-ESI easy to hand-roll.

Also, short lived processes may have been spawned by long lived processes, so changing how they're created can be done without taking anything down.

I find this style so much more performant than the one thread per request thing from Ruby, Perl, PHP et al. It's very easy and natural to minimize the amount of time spent in the db (which is where I'm usually bound.)

Re: The Erlang Shell

#57

Post by simula67 »

phamilton wrote: post #53
simula67 wrote: post #30

Earlier quoted context omitted.

If you have external state stored in say a database, I think you have to control your deployments to migrate the schema before the model objects can be deployed.

So no. One of the beauties of Erlang's hot code loading is the ability to migrate internal application state. An example: I'm storing a lookup table going from key to value as a tuple {key, value}. With a change going out, I'm going to start doing {key, {value, other-value}}. In Erlang there's an easy pattern in place to do that. Just provide a function that goes from {key, value} to {key, {value, sensible-default}}…

You may have misunderstood. A file can have multiple, say 'deployment units'. You can have a 'migrate deployment unit' which depend on the 'model deployment unit' which forces the app server to deploy the migrate unit first. You can have code there that migrates the tables and fill with default values before your model entities become available for use. From your comment this seems to be the same as what Erlang does for your example, unless you are claiming that Erlang's implementation is superior which could be the true, since I have never written any non-trivial programs in it.

Re: The Erlang Shell

#58

Post by davidw »

klibertp wrote: post #47
e_proxus wrote: post #35

Earlier quoted context omitted.

If you want to develop web pages, don't do Erlang. If you want to do web services, do Erlang. I've used both Ruby (Rails) and Python (trying to do concurrency with tornado, greenlets, gevent) and they just fell flat. You can do it, but it is incredibly messy and unstable (because there's so much work involved). In Erlang, you would typically answer a web request in a new actor every time. Just as in Rails, running fo…

> If you want to develop web pages, don't do Erlang. I'm not sure I agree. There's https://github.com/ChicagoBoss/ChicagoBoss and quite a few other frameworks ( https://github.com/ChicagoBoss/ChicagoBoss/wiki/Comparison-o... ) and there's also Elixir and some frameworks for it ( https://github.com/dynamo/dynamo ). It's of course nowhere near the ease of development of simple web pages in Django or Rails, but it's not…

> it's not like you need to reinvent everything from scratch either

No, not everything, but lots of stuff.

Recent example: I had to fix the Postgres database driver to better handle queries like

    "select * from foobar where id = any($1)",
      [uuid1, uuid2, uuid3]
Also, ChicagoBoss is a bit rudderless at this point in time - Evan Miller has moved on to other things, and the guy who briefly took his place has as well. My own branch of it on github seems to be the only one getting any attention lately, and that's just small fixes here and there:

https://github.com/davidw/ChicagoBoss

Re: The Erlang Shell

#59

Post by asabil »

chubot wrote: post #34
e_proxus wrote: post #32

Earlier quoted context omitted.

I read your discussion and really get all the points, and still I just think "don't make me think." The thing with Erlang is, I don't have to care about threads, pointers, libraries, re-entrant code, thread safety etc. I can just write a couple of actors, who do one thing each (in parallel, because Erlang) and they can talk to each other, share some data if they want. The only thing I need to keep in mind is, who has…

Sure, I wasn't saying not to use Erlang. Just saying that it is very possible to use threads and pointers in a sane (and Erlang-style) way. I am a big fan of interpreted languages and use them as my default. The situation I was describing was basically the only time I've ever rewritten in C++ for speed! It just doesn't happen that often. But I do wish the interpreted languages like Python, node.js, and Erlang all had…

It wouldn't make sense to have a network actor and a parsing actor. Because these two tasks (downloading the data and parsing it) are not concurrent, you should instead have a crawler actor that downloads -> parses -> stores.

Re: The Erlang Shell

#60

Post by phamilton »

pessimizer wrote: post #56
phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

>In Rails, a process is short-lived by design (the lifecycle of a request). I'm not sure if this is by design, or a limitation of Ruby. >short-lived processes aren't the best approach for a web service. Not for me - I'm building all of my web services in Erlang these days, and there's little to no relationship between the life of most processes in the service and the life of requests. Although each request is usually…

This is exactly the type of response I was looking for.

A few questions and comments:

1. RAM caching - I can see a big benefit to keeping cache in Erlang. No marshaling of data to and from memcached, for example. How do you handle eviction?

2. Avoiding the db - I'm going to guess this ties in with RAM caching. Are you doing a write through cache or something like that?

Thanks for your response, it's helping it all to come together for me.

Post reply on HN