Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Erlang Shell

medium.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: The Erlang Shell

#31

Post by chubot »

rdtsc wrote: post #27
chubot wrote: post #26

Earlier quoted context omitted.

Nitpick: mixing threads with pointers isn't really the problem; it's mixing threads with shared mutable state. A concurrency idiom I've gotten a lot of mileage out of is to use threads, without shared state, but with pointers. Instead of shared state, you use a message passing style where you pass pointers over queues. Threads only mutate data that they are explicitly handed via a queue. After putting an object on a…

That is true. One can build an almost shared nothing architecture in any of the languages supporting thread safe queues and threads. It is a continuum. One could write it with shared mutable state too, using locks. The problem is it is easy to mess up. C, a large program could load modules, and some of them are thread safe some are not. One could bring in a new module that calls some initialization routine (say curl_…

Yeah, but the beauty of this approach is that it always makes architectural sense to put a given library in a single stage, and in that stage only. That library is enclosed in an "actor", and it communicates with the other stages/threads via an application-specific message (not using data types from the library; you want the loose coupling).

If the library has non-reentrant code, then you can't have more than one thread for the stage -- you can't parallelize it. But it generally won't lead to correctness problems.

curl is actually a great example, because it has an event loop for parallelism (rather than threads). So you would run a single thread for curl, because you wouldn't want more than one thread anyway.

Say you are writing a web crawler. From the network/curl stage, you just pass off pointers to blocks of memory to parsing threads. Parsing threads will be CPU bound so you will likely want to run instances of that loop in multiple threads, and you will be able to do it with no problem, since they don't depend on the curl library. They just take in blocks of memory and output some data structure to another queue.

This is also a good way to compose say the curl event loop with event loops from other libraries (GUI libraries, perhaps). Hence the relation to SEDA (http://en.wikipedia.org/wiki/Staged_event-driven_architectur...).

Re: The Erlang Shell

#32

Post by e_proxus »

chubot wrote: post #31
rdtsc wrote: post #27

Earlier quoted context omitted.

That is true. One can build an almost shared nothing architecture in any of the languages supporting thread safe queues and threads. It is a continuum. One could write it with shared mutable state too, using locks. The problem is it is easy to mess up. C, a large program could load modules, and some of them are thread safe some are not. One could bring in a new module that calls some initialization routine (say curl_…

Yeah, but the beauty of this approach is that it always makes architectural sense to put a given library in a single stage, and in that stage only. That library is enclosed in an "actor", and it communicates with the other stages/threads via an application-specific message (not using data types from the library; you want the loose coupling). If the library has non-reentrant code, then you can't have more than one thr…

I read your discussion and really get all the points, and still I just think "don't make me think." The thing with Erlang is, I don't have to care about threads, pointers, libraries, re-entrant code, thread safety etc. I can just write a couple of actors, who do one thing each (in parallel, because Erlang) and they can talk to each other, share some data if they want. The only thing I need to keep in mind is, who has what data (if I need it).

I can hook up to the Erlang runtime (in local, staging, production, wherever) and talk to these actors. I can ask them "hey, what's your state now," "who are you talking to?" Sure, Erlang is slower in some cases, but for the value you get, I think it is priceless in many cases.

Re: The Erlang Shell

#33

Post by alcari »

shanselman wrote: post #22
biokoda wrote: post #20

Earlier quoted context omitted.

When you are running in production and something is not right, you can attach to the console and see what is going on. This is of course not something that you should do often, but can be a real life saver. Simulating something that is designed to work with thousands or even millions of users at the same time is pretty much impossible. Hot code loading and attaching to console gives you the option to see what is goin…

Why should you not do it often?

If you find yourself often having to manually access the console on a production system, clearly something in your testing and deployment methodology is horribly wrong.

Re: The Erlang Shell

#34

Post by chubot »

e_proxus wrote: post #32
chubot wrote: post #31

Earlier quoted context omitted.

Yeah, but the beauty of this approach is that it always makes architectural sense to put a given library in a single stage, and in that stage only. That library is enclosed in an "actor", and it communicates with the other stages/threads via an application-specific message (not using data types from the library; you want the loose coupling). If the library has non-reentrant code, then you can't have more than one thr…

I read your discussion and really get all the points, and still I just think "don't make me think." The thing with Erlang is, I don't have to care about threads, pointers, libraries, re-entrant code, thread safety etc. I can just write a couple of actors, who do one thing each (in parallel, because Erlang) and they can talk to each other, share some data if they want. The only thing I need to keep in mind is, who has…

Sure, I wasn't saying not to use Erlang. Just saying that it is very possible to use threads and pointers in a sane (and Erlang-style) way.

I am a big fan of interpreted languages and use them as my default. The situation I was describing was basically the only time I've ever rewritten in C++ for speed! It just doesn't happen that often.

But I do wish the interpreted languages like Python, node.js, and Erlang all had better and more consistent C APIs (more like Lua's). C itself is not that hard, but the C APIs definitely put people off.

Actually I wonder for Erlang, with the web crawler example, would you have to copy entire web pages if you wanted to pass them off from a "network actor" to a "parsing actor"? The threads + pointers solution easily avoids that, while retaining modularity.

Re: The Erlang Shell

#35

Post by e_proxus »

phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

If you want to develop web pages, don't do Erlang. If you want to do web services, do Erlang. I've used both Ruby (Rails) and Python (trying to do concurrency with tornado, greenlets, gevent) and they just fell flat. You can do it, but it is incredibly messy and unstable (because there's so much work involved).

In Erlang, you would typically answer a web request in a new actor every time. Just as in Rails, running for example N number of threads serving one request each. Of course, in Erlang that number of actors can vary, which means your system will be more responsive and you'll have lower average latency. However, what differs mostly, is that in Erlang all those actors live inside one VM and you can have central actors managing things, collecting statistics, keeping global state, talking to databases / other services. This, I think, is the core strength of Erlang compared conventional single threaded / GIL systems.

A more concrete example would be to dynamically show the current request rate on a web page. You can just create an actor that gets a message from any other actor once they handle a request, and keeps track of how many such messages per second arrive. Then when rendering that web page, you just ask that actor for it's current value. Simple, beautiful, dynamic. No need for a database or any other central storage.

Re: The Erlang Shell

#36

Post by e_proxus »

chubot wrote: post #34
e_proxus wrote: post #32

Earlier quoted context omitted.

I read your discussion and really get all the points, and still I just think "don't make me think." The thing with Erlang is, I don't have to care about threads, pointers, libraries, re-entrant code, thread safety etc. I can just write a couple of actors, who do one thing each (in parallel, because Erlang) and they can talk to each other, share some data if they want. The only thing I need to keep in mind is, who has…

Sure, I wasn't saying not to use Erlang. Just saying that it is very possible to use threads and pointers in a sane (and Erlang-style) way. I am a big fan of interpreted languages and use them as my default. The situation I was describing was basically the only time I've ever rewritten in C++ for speed! It just doesn't happen that often. But I do wish the interpreted languages like Python, node.js, and Erlang all had…

Yes, Erlang has a "share nothing" concurrency model. You would need to send the data over to another process. There is one exception, which would come into play here, and that is binary data. Normal Erlang data ("Erlang terms") such as lists, tuples, strings, integers etc. are always copied, but binary data (raw binary data, but often representing string data) is just referenced. If you download a web page, you'd save the whole content as a binary, send it over to the other actor (and it would feel just as being "copied" because there is no difference compared to the other data types). Under the hood, Erlang would just pass the reference and handle everything for you (reference counting, garbage collection etc.).

Also, when you split, or reference sub binaries, these become just pointers into the original binary data.

Re: The Erlang Shell

#37

Post by davidw »

phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

> In Rails, a process is short-lived by design (the lifecycle of a request)

Yes and no. Rails processes are so huge that they are loaded up and kept around between requests. Erlang processes (which are not unix processes!) are much smaller and lighter. One Erlang program can handle tons of concurrent, always open requests, whereas Rails just isn't cut out for that.

As to what Erlang is good for, I think its sweet spot with web stuff is where you need a lot of always open connections, such as with web sockets.

I'm a fan of Erlang, but for most SaaS kinds of startuppy things, am not sure it's a good fit: you get so many more tools packaged up and easy to use for you with Rails that unless you know Rails does not work for you, I would almost always choose Rails.

Re: The Erlang Shell

#38

Post by Jacqued »

phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

Since you recently started out, may I ask you what good resources you found to get started with Erlang ?

Re: The Erlang Shell

#39

Post by cmsd2 »

simula67 wrote: post #15
agentultra wrote: post #12

Earlier quoted context omitted.

> Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? Yes. > I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang. It's not. iIRC there's even a version of the JVM with code reloading. What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definit…

> there's even a version of the JVM with code reloading. Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.

sure, any existing connections continue to be serviced by the old code, and new connections are picked up by the new code.

if you need more elaborate lifecycle management of components as they are upgraded, then maybe OSGI could do that.

erlang's approach really appeals to me though -- not only does the code get upgraded, but it's in cooperation with its message loop so you don't miss a step, and you also get a callback to migrate your actor state.

Re: The Erlang Shell

#40

Post by davidw »

Jacqued wrote: post #38
phamilton wrote: post #13

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). A…

Since you recently started out, may I ask you what good resources you found to get started with Erlang ?

This is a good book to start with:

http://learnyousomeerlang.com/

Free on the web, but buy a copy - it's worth it.

Post reply on HN