Live data from Hacker News

What can I only do in Erlang?

erlang.org

161–170 of 173 posts

Re: What can I only do in Erlang?

#161
post #107

I'm just starting to do some work on a system that is going to be very distributed with many endpoints, often on unreliable networks connecting back to a server. Erlang (elixir, really) looked promising for this so I've been investigating off and on while we do some prototyping and flesh out some of the details. So far it seems to be that you get nearly all the advantages of Erlang with running something like RabbitM…

RabbitMQ is a good piece of software, but it has issues. You mention unreliable connections; RMQ does not handle that well. Even just a few seconds of network hiccup will throw it into a partitioned state, out of which you cannot reliably escape without manual intervention. There are also quite a few bugs that you tend to hit only when RMQ is exposed to network issues. To use RMQ with bad networks, you need to use ei…

Man, I hated to read that. Thankfully I haven't done anything more than just an hour or so of playing with it, but nothing I've read indicated there were issues with connectivity :(

Could you point me to or elaborate on some specifics on the partitioned states and manual interventions required? I'm only really calling the shots on the server side and endpoints will be written in various languages, part of my reason for looking at RabbitMQ since it has so many libs.

Re: What can I only do in Erlang?

#162
post #9

In 2005, I worked in software development for 6 months before abandoning it for another field. In my new job, I was given the task of writing a server for a messaging application - which would allow users to send a "hand-drawn" message from our own proprietary handheld device to Windows phones. I was told to learn Erlang and get it done - the company was barely 30 people and had no formal training programs. While I h…

You can always play with Elixir. http://elixir-lang.org/

Re: What can I only do in Erlang?

#163
post #161

Earlier quoted context omitted.

RabbitMQ is a good piece of software, but it has issues. You mention unreliable connections; RMQ does not handle that well. Even just a few seconds of network hiccup will throw it into a partitioned state, out of which you cannot reliably escape without manual intervention. There are also quite a few bugs that you tend to hit only when RMQ is exposed to network issues. To use RMQ with bad networks, you need to use ei…

Man, I hated to read that. Thankfully I haven't done anything more than just an hour or so of playing with it, but nothing I've read indicated there were issues with connectivity :( Could you point me to or elaborate on some specifics on the partitioned states and manual interventions required? I'm only really calling the shots on the server side and endpoints will be written in various languages, part of my reason f…

You may want to read Aphyr's analysis [1], which is rigorous and well written.

If you read the RabbitMQ documentation, they readily admit to the fact that the clustering support is not designed for unreliable network connections. It's mentioned only once, and it's easy to overlook.

Here's a very quick, rough overview (it's complicated):

RabbitMQ works by assigning queues to specific nodes. A queue is a physical thing belonging to only one node at a time; consumers, publishers and so on are distributed, but queues aren't. If you lose that node, you lose the queue. If you get a network partition, the nodes in the other partition(s) will not see the queue anymore. Connected publishers and consumers will start barfing when their bindings no longer seem to exist.

RabbitMQ can mitigate these loss scenarios by mirroring queues. They call this "high availability", or HA [2]. Each queue will have a master node and a number of slave nodes that maintain replicas. In the event of a cluster issue, a slave will be elected master.

Ironically, HA, despite its name and apparent purpose, still requires a stable network. It does not deal well with network partitions. It's not really high availability. It is, I suspect, designed for situations when you wanted to take down a node manually, in a controlled manned, without disturbing a live cluster.

The reason, and this is the important part, is that when a network issue is resolved and the partitions can talk to each other again, RabbitMQ has no way of reconciling queues. If the partition divided nodes into sets A and B, then A will likely have promoted a mirror slave to a master, while B will have kept its master, or vice versa. When the partition is resolved, you now have two masters, whose queues have very likely gotten out of sync in the mean time.

When this happens, RabbitMQ normally needs to be manually restored; it will simply cease to work properly. Note that I'm talking about the case when the network issue has been resolved. The network is back, RabbitMQ isn't. You need to decide which nodes should survive and which should be wiped, and this recovery process is manual. Of course, if this happens in the middle of the night, you have a potentional emergency.

Fortunately, RabbitMQ has a band aid on top of HA called "automatic partition handling" [3]. It offers three different modes of behaviour, of which autoheal is probably the most important one. In autoheal mode, RabbitMQ will automatically pick a winner to solve conflicts, and promote it to a master. This will result in data loss; the losing nodes will be wiped. Autoheal is actually pretty crazy, because it is lossy by design; the only way to find out if you lost any data today is by grepping the logs for autoheal messages.

Worse, due to the "poor man's database" nature of RabbitMQ, doing any kind of manual recovery -- like dumping the conflicts, reconciling them, and dumping them back in -- is probably not realistic when it happens.

Note that even on a flawless LAN, RabbitMQ can get network partitions if your server has high enough CPU or I/O load. Another thing that can induce network partitions is soft kernel lockups, which is common enough in virtualized environments; for example, VMware's vMotion can move a node from one physical machine to another, a process which can take several seconds, during which time RabbitMQ gets a fit. In such cases you'll want to increase the "net_tick_timeout" [4] setting.

[1] http://aphyr.com/posts/315-call-me-maybe-rabbitmq

[2] https://www.rabbitmq.com/ha.html

[3] https://www.rabbitmq.com/partitions.html#automatic-handling

[4] https://www.rabbitmq.com/nettick.html

Re: What can I only do in Erlang?

#164
post #34

Erlang looks interesting. However, the only problem I have with starting in Erlang is that on the great computer language shootout, it shows that Erlang is about 10x slower than C++ on most examples. And about 3x slower than Go [1] I know that the problems on this website are not specifically "concurrent" problems, but still, even a distributed web-server must do some non-concurrent stuff at times :) Are my concerns…

I think you raised valid concerns about the single node performance of Erlang. If you have a single node problem or you are looking for raw speed Erlang is not the best option.

The fun fact is that most distributed systems don't really care about 7x slower too much. Those care about scalability, reliability and fault tolerance more. One of the most reliable software systems out there was written in Erlang and as far as I know, there are few system can match that sort of uptime[1]. There are few large scale systems in Erlang, including one Amazon AWS service[2], and bunch of others like WatsApp.

I also like to mention the VM that is pretty amazing. There is per process (not an OS process) garbage collection that enables Erlang to meet with super tight SLAs in terms of latency. Ideal for high scale websites, doing the routing, but also for moving data from A to B. The concurrency model is also one of the strongest features of Erlang. Message passing is very powerful and enables you writing asynchronous code that is still easy to read and follow. Go can match that probably with channels and go routines though.

If you care about single node performance you can either use NIFs[3] or if you not tied to Erlang chose a language that is focusing on that more.

1. http://ll2.ai.mit.edu/talks/armstrong.pdf 2. http://web.archive.org/web/20110623221347/http://www.satine.... 3. http://www.erlang.org/doc/tutorial/nif.html

Re: What can I only do in Erlang?

#165
post #87

Earlier quoted context omitted.

Down voting is not for disagreeing, read the guidelines for HN. If you disagree, provide a comment as to why. Down voting is for completely irrelevant comments, inappropriate comments, etc... [EDIT] To be clear, I did not downvote you and I disagree with the down votes on your comment here, even though it's off-topic it's important to acculturate people that are accustomed to "downvoting for disagreeing". Which is wh…

Downvoting is for however the user wants to use it. https://news.ycombinator.com/item?id=117171 > pg 2455 days ago | link | parent | flag > I think it's ok to use the up and down arrows to express agreement. Obviously the uparrows aren't only for applauding politeness, so it seems reasonable that the downarrows aren't only for booing rudeness. > It only becomes abuse when people resort to karma bombing: downvoting a…

Thank you! HN is not Reddit, and I'm happy down vote can be used as a "I don't agree with you button".

Re: What can I only do in Erlang?

#166
post #87

Earlier quoted context omitted.

Down voting is not for disagreeing, read the guidelines for HN. If you disagree, provide a comment as to why. Down voting is for completely irrelevant comments, inappropriate comments, etc... [EDIT] To be clear, I did not downvote you and I disagree with the down votes on your comment here, even though it's off-topic it's important to acculturate people that are accustomed to "downvoting for disagreeing". Which is wh…

Downvoting is for however the user wants to use it. https://news.ycombinator.com/item?id=117171 > pg 2455 days ago | link | parent | flag > I think it's ok to use the up and down arrows to express agreement. Obviously the uparrows aren't only for applauding politeness, so it seems reasonable that the downarrows aren't only for booing rudeness. > It only becomes abuse when people resort to karma bombing: downvoting a…

In that case I stand corrected - I thought I remembered reading that in the guidelines.

Downvotes instead of comments is still a personal peeve but I guess I can't chide people for it on HN, lol.

Re: What can I only do in Erlang?

#167
post #51

Earlier quoted context omitted.

Erlang's strength is in its virtual machine which has supervision trees built-in, a built-in DNS system, built-in inter-node RPC, hot code reloading, massive concurrency, etc... I still haven't experienced a programming environment that replicates the Erlang/OTP environment. Period. Haskell is amazing , I've built my company on it, but we're also building our company on Erlang - Haskell's tooling around distributed c…

out of curiosity, what's your company and/or field?

http://plumlife.com

It's a fun product :)

Re: What can I only do in Erlang?

#168

Curious how much people think Go will eat into Erlang? Many of the points made were about Erlang as CSP and that's Go-territory. Loïc Hogun (author of Cowboy, other projects) said: For me Erlang is first fault tolerant, then concurrent, then functional As Go gains libraries like groupcache will it become more and more go-to for networked and shared systems? Is Go moving up from concurrent to fault-tolerant in a real…

Having used both, I don't think Go and Erlang are even close in comparison. Go's lack of go routine isolation and error handling will never allow it to properly handle the problems Erlang was designed for without a lot of defensive programming and pain. Start 20,000->30,000 go routines and have one of them crash, taking down the other 29,999 go routines and now try and debug that problem in production? Go tries to advertise that it was built writing important services in, but it's shared everything model really argues against that. Erlang's hot code loading, runtime inspection, production quality REPL, and process isolation makes it a completely different language than Go. Until Go routines and channels get way way better, I would never attempt to write a distributed server that needs high availability in Go, it is just too fragile atm.

On the other side of the coin, if I wanted to write a simple command line tool quickly that was easy to deploy to many servers, Go might be a great choice, where Erlang is a bit painful for small and simple programs.

Re: What can I only do in Erlang?

#169

erlang's whole design is what OO promised but failed to deliver. it's completely trivial and natural to represent programs as a collection of loosely coupled state machines that communicate asynchronously. it turns out this is a fantastic model for anything network related but also for programming in general

This is quite interesting. I am currently working on a Java based state machine and it solves problems in a very unique way. It is quite a change from procedural code.

Re: What can I only do in Erlang?

#170
post #34

Erlang looks interesting. However, the only problem I have with starting in Erlang is that on the great computer language shootout, it shows that Erlang is about 10x slower than C++ on most examples. And about 3x slower than Go [1] I know that the problems on this website are not specifically "concurrent" problems, but still, even a distributed web-server must do some non-concurrent stuff at times :) Are my concerns…

Erlang lets you write maintainable, robust code.

C++ lets you write fast code.

Haskell is interesting.

If you want maintainable AND fast, have a look at Rust.

Post reply on HN