Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Erlang/OTP by Example

erlangbyexample.org

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Erlang/OTP by Example

#31

Post by jhgg »

ilovecaching wrote: post #25
plainOldText wrote: post #23

Earlier quoted context omitted.

For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read. At t…

I content Erlang can be very hard to reason about. I've seen some extremely gnarly Erlang where the author didn't write a -spec and it was almost impossible to tell what shape a tuple parameter would take. In Rust the compiler enforces all of this for you. Rust gives you: - Maintainability: Rust signatures not only tell you the types of arguments, but also their lifetimes. A signature in Rust is an extremely strong a…

Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does.

Coming from a company that uses elixir heavily and has made a significant investment in rust, I don’t think we would ever use solely rust on our distributed systems. However, we have rewritten some code that elixir was too slow at in rust and exposed it as a NIF on BEAM - and that has worked well. (Blog post on that soon hopefully).

I do admit, we are also going to be ditching mnesia for one of our clusters for our own in-house simpler system (ETS replication with different consistency/netsplit guarantees for our use-case), we've had to write our own cross-node process monitoring solution (at peak we see 200M+ cross-node monitors on our cluster), and we've also had to overcome the limits of message-fanout on distribution as well (https://github.com/discordapp/manifold).

However, for operating at our scale (peak 9m ccu @ 5m events/sec fanout to clients), we run a surprisingly small number of servers for our real time system (~120).

EDIT: I can't reply to your post below, but I think the runtime introspection we run into is not dealing with OS level metrics, but application level introspection. Introspecting the state of processes, writing code in the repl to debug issues within the cluster, benchmarking to find hot functions or where specific processes are spending a lot of their time. Capturing traffic to replay on a test cluster to simulate production load, all becomes very trivial with BEAM.

Re: Erlang/OTP by Example

#32

Post by tpetry »

ilovecaching wrote: post #19
plainOldText wrote: post #17

Earlier quoted context omitted.

Rust has too big of a learning curve, and it's quite complex for writing service oriented systems. Granted, having a static compiler eliminates many types of bugs found in other systems. In my opinion a more sensible approach is Elixir + Rust (via rustler). Your get the elegance and productivity of Elixir, the concurrency and fault tolerance of Erlang/OTP, while still being able to writing super fast, low level code…

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

You are comparing the learning curve of a complete ecosystem (erlang wirh clustering) to a barebone programming language (rust). Isn‘t this totally unfair? ;) Learning another framework for rust (if it exists) to solve the same problems as erlang (transparent clustering of everything) will add many hours too.

Re: Erlang/OTP by Example

#33

Post by ilovecaching »

andy_ppp wrote: post #29
ilovecaching wrote: post #5

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be…

You’re comparing the out of the box, network transparent scaling solution that is fully meshed and doesn’t claim to solve everyone’s problems forever with a solution in rust that probably involves a completely hand rolled implementation of lots and lots of different things OTP is doing. You have made me want to learn Rust though, how does it compare with Golang?

We move the orchestration from the Erlang runtime to our container orchestrator though. We have a better separation of concerns, and we can scale any language. The registry, process communication, are just services and protocols running in containers.

So compared to Go, the Rust folks seemed to have paid close attention to the things people liked about Go. We have most of those things. We have great tooling, auto formatting, concurrency primitives (although we have freedom of choice. Green threads are a third party library, and aren't part of a runtime). We even have an animal mascot.

From a language standpoint, Go isn't anything remarkable (our Haskell colleague calls it a step backwards). If you listen to the Gotime podcast, Brad Fitzpatrick, one of the Go authors even says as much. Go's big contribution was essentially web programming and go routines.

Rust, on the other hand, has a lot. It has typeclasses from Haskell. It has lifetimes from the PL research community. It has immutability and pattern matching just like in Erlang. It has all the familiar control structure and enums/structs from C++.

Performance of Go in terms of CPU and memory pressure is usually somewhere near Java, if not slightly better. That's pretty damn good. The JVM has improved a lot over the years. Rust performance is a hair shy of C/C++. That's amazing. All those abstractions, and the speed of C++?

Re: Erlang/OTP by Example

#34

Post by jhgg »

jlouis wrote: post #3

If you get to the part about monitors, consider erlang:demonitor(Ref, [flush]) in your own code. This also removes the monitor message from your mailbox if it arrived in between, which is a real problem in an async setting. Though in some situations, it is better to just ignore the spurious message when it arrives by tracking what monitors you have enabled in the process state. Unkonwn monitors are just gracefully ig…

demonitor flush can be harmful in some instances as it invokes a selective receive to yank the down message from your process's mailbox. In the event that you are demonitor+flushing on a process that has a growing message queue, every flush gets more and more costly - especially if the monitor has already delivered a DOWN message for it to yank.

it's almost always better to ignore down messages that you don't care about.

Re: Erlang/OTP by Example

#35

Post by ilovecaching »

tpetry wrote: post #32
ilovecaching wrote: post #19

Earlier quoted context omitted.

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

You are comparing the learning curve of a complete ecosystem (erlang wirh clustering) to a barebone programming language (rust). Isn‘t this totally unfair? ;) Learning another framework for rust (if it exists) to solve the same problems as erlang (transparent clustering of everything) will add many hours too.

That is the language though. Erlang is DSL for writing massively concurrent, distributed programs. Rust is a general purpose language that enables systems programming. Learning how to do either one of those activities with their respective language is all factored into my assessment.

Re: Erlang/OTP by Example

#36

Post by ilovecaching »

jhgg wrote: post #31
ilovecaching wrote: post #25

Earlier quoted context omitted.

I content Erlang can be very hard to reason about. I've seen some extremely gnarly Erlang where the author didn't write a -spec and it was almost impossible to tell what shape a tuple parameter would take. In Rust the compiler enforces all of this for you. Rust gives you: - Maintainability: Rust signatures not only tell you the types of arguments, but also their lifetimes. A signature in Rust is an extremely strong a…

Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does. Coming from a company that uses elixir heavily and has made a significant investment in rust, I don’t think we would ever use solely rust on our distributed systems. However, we have rewritten some code that elixir was too slow at in rust and exposed it as a NIF on BEAM - and that has worked well. (Blog p…

We couldn't get enough introspection. Now we use Rust and we can use a much larger toolkit. We have BCC, kprobes, flamegraphs... When you use Rust the OS is your machine. With Elixir you need to understand the BEAM, a slightly esoteric platform with a rather small community working on it.

Re: Erlang/OTP by Example

#37

Post by sumnulu »

rhizome31 wrote: post #24
ilovecaching wrote: post #19

Earlier quoted context omitted.

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

Would you say that learning Erlang/OTP has given you transferable knowledge about building concurrent and distributed system? Did you transfer any of that knowledge to your Rust designs? Do you have an opinion about Scala/Akka?

Very happy with Akka. It is fast (thanks to JVM), has lots of build-in building blocks / modules and documentation covers everything, also community is very helpful.

https://akka.io/docs/

Re: Erlang/OTP by Example

#38

Post by ilovecaching »

dev_dull wrote: post #30
ilovecaching wrote: post #13

Earlier quoted context omitted.

In my experience any compiled language is easier to manage. There is no disconnect between your runtime and the system. You're investing in systems knowledge rather than knowledge of a particular VM's bytecode/abstractions as well. Erlang does provide good solutions to lifecycle management. The problem is that whenever someone picks a solution for you, you're now locked it and it can be difficult to migrate to a new…

Totally agree regarding compiled languages. I’m surprised python et all don’t offer a way to check for file/script “correctness” without execution. It’s so bad that I will often add a “-h” flag to scripts I encounter just so I can run it and make sure no one introduce a silly bug (such as typo, import, etc).

Yep, I've seen my fair share of Python stack traces. Completely sympathize, and it is one of the reasons I believe in statically types and compiled as well.

Re: Erlang/OTP by Example

#39

Post by Immortalin »

ilovecaching wrote: post #5

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be…

There's always Pony.

Re: Erlang/OTP by Example

#40

Post by tomsmeding »

ilovecaching wrote: post #25
plainOldText wrote: post #23

Earlier quoted context omitted.

For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read. At t…

I content Erlang can be very hard to reason about. I've seen some extremely gnarly Erlang where the author didn't write a -spec and it was almost impossible to tell what shape a tuple parameter would take. In Rust the compiler enforces all of this for you. Rust gives you: - Maintainability: Rust signatures not only tell you the types of arguments, but also their lifetimes. A signature in Rust is an extremely strong a…

> and enforces exhaustion. Not even Haskell does that.

Maybe it doesn't throw an error message, but ghc does warn about it when you use the -Wall flag. I believe you can get the behaviour you want by turning on only that specific warning (I forget what it's called) and using -Werror.

Post reply on HN