Erlang/OTP by Example • Hacker News Board

Re: Erlang/OTP by Example

#21

Post by ilovecaching » Sat, Oct 20, 2018, 9:17 PM UTC

bthornbury wrote: post #16

ilovecaching wrote: post #15

Earlier quoted context omitted.

I'm aware of all of these claims. Any Erlang developer has heard this shtick a thousand times. The reality is that you do eventually have to twist the BEAM once you reach a certain level of scale, and there are plenty of bugs. Like I said, it's also difficult to burst, so it's not always as black and white as you say. It's also missing a lot of the introspection you need to operationalize at scale. We've had to patch…

As an Erlang fan, who hasn't used it in production services, I'm wondering if you can let us know some specific scaling issues you encountered.

I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.

Re: Erlang/OTP by Example

#22

Post by zinclozenge » Sat, Oct 20, 2018, 9:21 PM UTC

ilovecaching wrote: post #15

enraged_camel wrote: post #12

Earlier quoted context omitted.

Your complaints about Erlang give me the impression that you may have picked the wrong language for your problem domain, and paid for that mistake by having to do things like rewriting gen_server. Erlang was designed to be reliable and scalable . It was not designed to be fast . If fast is a hard prerequisite for you, you're right, you should go with another language. But Erlang also does a ton of things correctly (a…

I'm aware of all of these claims. Any Erlang developer has heard this shtick a thousand times. The reality is that you do eventually have to twist the BEAM once you reach a certain level of scale, and there are plenty of bugs. Like I said, it's also difficult to burst, so it's not always as black and white as you say. It's also missing a lot of the introspection you need to operationalize at scale. We've had to patch…

This is exactly what whatsapp experienced, but with more recent OTP versions I would have assumed that those went away and the main problem once you reach 'scale' is that the distributed erlang network topology gets saturated. I'm curious what difficulties you ran into?

edit: nevermind I saw your other post, but confirms what I expected.

Re: Erlang/OTP by Example

#23

Post by plainOldText » Sat, Oct 20, 2018, 9:21 PM UTC

ilovecaching wrote: post #19

plainOldText wrote: post #17

Earlier quoted context omitted.

Rust has too big of a learning curve, and it's quite complex for writing service oriented systems. Granted, having a static compiler eliminates many types of bugs found in other systems. In my opinion a more sensible approach is Elixir + Rust (via rustler). Your get the elegance and productivity of Elixir, the concurrency and fault tolerance of Erlang/OTP, while still being able to writing super fast, low level code…

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read.

At the end of the day, use whatever language you prefer, just keep in mind that software needs to be (1) shipped, (2) maintained (by multiple people who read each other's code) and (3) evolved. I would also add (0) experimentation; before one even ships any code, one ought to easily experiment with various ideas.

Re: Erlang/OTP by Example

#24

Post by rhizome31 » Sat, Oct 20, 2018, 9:27 PM UTC

ilovecaching wrote: post #19

plainOldText wrote: post #17

Earlier quoted context omitted.

Rust has too big of a learning curve, and it's quite complex for writing service oriented systems. Granted, having a static compiler eliminates many types of bugs found in other systems. In my opinion a more sensible approach is Elixir + Rust (via rustler). Your get the elegance and productivity of Elixir, the concurrency and fault tolerance of Erlang/OTP, while still being able to writing super fast, low level code…

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

Would you say that learning Erlang/OTP has given you transferable knowledge about building concurrent and distributed system? Did you transfer any of that knowledge to your Rust designs?

Do you have an opinion about Scala/Akka?

Re: Erlang/OTP by Example

#25

Post by ilovecaching » Sat, Oct 20, 2018, 9:31 PM UTC

plainOldText wrote: post #23

ilovecaching wrote: post #19

Earlier quoted context omitted.

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read. At t…

I content Erlang can be very hard to reason about. I've seen some extremely gnarly Erlang where the author didn't write a -spec and it was almost impossible to tell what shape a tuple parameter would take. In Rust the compiler enforces all of this for you. Rust gives you:

- Maintainability: Rust signatures not only tell you the types of arguments, but also their lifetimes. A signature in Rust is an extremely strong abstraction. Traits can further constrain types making it essentially a game of plugging the right blocks in to the right holes. Erlang is like having the blocks but all the holes are under a tablecloth and you just have to guess how to fit them in.

- Correctness: Without a static type system Erlang does very little for correctness. Rust has pattern matching as well and can enforces exhaustion. Not even Haskell does that. Lifetimes guarantee safety even for shared pieces of data.

- Speed: Obviously, Rust is a compiled, manually memory managed language with near C++ performance.

On every metric, Rust offers a lot more than Erlang. As someone who has spent years with Erlang, Rust is simply better for professional development.

Re: Erlang/OTP by Example

#26

Post by ilovecaching » Sat, Oct 20, 2018, 9:37 PM UTC

rhizome31 wrote: post #24

ilovecaching wrote: post #19

Earlier quoted context omitted.

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

Would you say that learning Erlang/OTP has given you transferable knowledge about building concurrent and distributed system? Did you transfer any of that knowledge to your Rust designs? Do you have an opinion about Scala/Akka?

I don't have experience with Scala/Akka. Erlang is a good language to learn clustering/concurrency, SMP, queuing theory, etc. Absolutely. If you learn about concurrency through Erlang, you'll have the right mindset. But you should also read Simon Marlow's book about concurrency in Haskell. You should learn pthreads and futexs so you can understand the building blocks of mailboxs and channels and higher level patterns.

I am a big believer that you should continually invest in learning and mastering new languages. Each language gives you a different perspective on how to solve problems.

But as far as the right tool to build professional software, there are caveats. Safety, correctness, speed, and maintainability are all important factors in choosing the right tools.

Edit: I should also say I've had plenty of interviewees who couldn't explain the difference between concurrency vs parallelism, and whose knowledge of these concepts was limited to spawning a pthread and locking shared data. Needless to say these people tend to do really poorly at explaining how to scale a distributed system.

Contrast that with the Erlang developers we interviewed, who think in terms of scale. Their answer is almost always something that could accept 10 requests/s or 10 million. Same thing with our Haskell interviewees. They'll answer our algorithms questions by writing out the types and deriving an answers with a single expression. I had an interviewer who was extremely confused by this. We hired the guy that confused him, not only does he understand concurrency, he also understands lazyness and we love lazy developers :)

Re: Erlang/OTP by Example

#27

Post by ilovecaching » Sat, Oct 20, 2018, 9:39 PM UTC

zinclozenge wrote: post #22

ilovecaching wrote: post #15

Earlier quoted context omitted.

I'm aware of all of these claims. Any Erlang developer has heard this shtick a thousand times. The reality is that you do eventually have to twist the BEAM once you reach a certain level of scale, and there are plenty of bugs. Like I said, it's also difficult to burst, so it's not always as black and white as you say. It's also missing a lot of the introspection you need to operationalize at scale. We've had to patch…

This is exactly what whatsapp experienced, but with more recent OTP versions I would have assumed that those went away and the main problem once you reach 'scale' is that the distributed erlang network topology gets saturated. I'm curious what difficulties you ran into? edit: nevermind I saw your other post, but confirms what I expected.

Correct

Re: Erlang/OTP by Example

#28

Post by ilovecaching » Sat, Oct 20, 2018, 9:40 PM UTC

pera wrote: post #14

ilovecaching wrote: post #11

Earlier quoted context omitted.

Elixir doesn't affect most of the operational aspects of Erlang, and it certainly in no way affects performance. You're still running on the BEAM, except now you actually have more apps running (The Elixir runtime runs as an app). Syntactically it cleans some things up, adds some niceties, and annoyingly makes atoms require a prefix of ":" (annoying because atoms are central to Erlang's readability). Personally, I do…

Are you actually using Rust for professional web development? I've tried it earlier this year and my experience wasn't that great, so I'm quite surprised to read this comment.

What issues did you have? We haven't had very many issues. We rebuilt some internals tools, but for the most part it's been smooth sailing. We do use hyper and h2.

Re: Erlang/OTP by Example

#29

Post by andy_ppp » Sat, Oct 20, 2018, 9:46 PM UTC

ilovecaching wrote: post #5

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be…

You’re comparing the out of the box, network transparent scaling solution that is fully meshed and doesn’t claim to solve everyone’s problems forever with a solution in rust that probably involves a completely hand rolled implementation of lots and lots of different things OTP is doing. You have made me want to learn Rust though, how does it compare with Golang?

Re: Erlang/OTP by Example

#30

Post by dev_dull » Sat, Oct 20, 2018, 9:49 PM UTC

ilovecaching wrote: post #13

dev_dull wrote: post #6

Earlier quoted context omitted.

The world of ephemeral containers really has changed the dynamic. Erlang solves a lot of problems with good solutions, but writing good software is only part of the battle. Lifecycle management, monitoring, etc make up a huge operational burden and should always be thought of as first-class problems in language design. Rust makes that better in many ways but not uniquely better than alternative languages.

In my experience any compiled language is easier to manage. There is no disconnect between your runtime and the system. You're investing in systems knowledge rather than knowledge of a particular VM's bytecode/abstractions as well. Erlang does provide good solutions to lifecycle management. The problem is that whenever someone picks a solution for you, you're now locked it and it can be difficult to migrate to a new…

Totally agree regarding compiled languages. I’m surprised python et all don’t offer a way to check for file/script “correctness” without execution. It’s so bad that I will often add a “-h” flag to scripts I encounter just so I can run it and make sure no one introduce a silly bug (such as typo, import, etc).