Live data from Hacker News

Erlang/OTP by Example

erlangbyexample.org

71–80 of 108 posts

Re: Erlang/OTP by Example

#71
post #55

Earlier quoted context omitted.

I'm going to disagree with you on Rust. It's a very different, very verbose language. It doesn't have any of the stories around immutability that Erlang does. You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. A common error in event-driven languages is th…

> I'm going to disagree with you on Rust. It's a very different, very verbose language. Erlang is incredibly verbose. It has very few abstractions. Rust has a lot of abstractions. I've rewritten a few Erlang projects now and Rust and I've been able to come out with close to or under the same LOC (I always use specs though). > It doesn't have any of the stories around immutability that Erlang does. let expressions are…

If you don't mind, can you give some more details on what you don't like about mnesia? Everybody says it's awful, and it surely has warts, but I've seen it scale ok to 512GB+ datasets in disc_copies tables, and it works alright with a couple caveats.

a) Not sure if it changed, but mnesia startup was very brain damaged -- fixing up the local data from disc, then throwing it all away to load from peers is a lot of wasted time. It's much faster to remove all the local tables on disk before starting the node so it short circuits to copying from the peers. Even when it's faster, sending over half a terrabyte of data takes a while. Some sort of persistent transaction logs for peers would be nice.

b) network partitions aren't fun at all

c) we direct mnesia read and write for a key into a specific process to enforce serialization, and then we use dirty read/writes; so we skip all the locking.

d) we've certainly patched a lot of things in transaction sending and receiving over long distances, especially needed if your network isn't clean.

Anyway, thanks for your thoughts all over this thread.

Re: Erlang/OTP by Example

#72
post #55

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be…

I'm going to disagree with you on Rust. It's a very different, very verbose language. It doesn't have any of the stories around immutability that Erlang does. You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. A common error in event-driven languages is th…

disterl is definitely worse than writing your own ipc. just use a language that has good sctp or h2 support and something like gRPC for the application layer. for peer discovery you have consul, etcd, zookeeper...

Re: Erlang/OTP by Example

#73
post #55

Earlier quoted context omitted.

I'm going to disagree with you on Rust. It's a very different, very verbose language. It doesn't have any of the stories around immutability that Erlang does. You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. A common error in event-driven languages is th…

disterl is definitely worse than writing your own ipc. just use a language that has good sctp or h2 support and something like gRPC for the application layer. for peer discovery you have consul, etcd, zookeeper...

That's essentially what we're doing now.

Re: Erlang/OTP by Example

#74
post #71

Earlier quoted context omitted.

> I'm going to disagree with you on Rust. It's a very different, very verbose language. Erlang is incredibly verbose. It has very few abstractions. Rust has a lot of abstractions. I've rewritten a few Erlang projects now and Rust and I've been able to come out with close to or under the same LOC (I always use specs though). > It doesn't have any of the stories around immutability that Erlang does. let expressions are…

If you don't mind, can you give some more details on what you don't like about mnesia? Everybody says it's awful, and it surely has warts, but I've seen it scale ok to 512GB+ datasets in disc_copies tables, and it works alright with a couple caveats. a) Not sure if it changed, but mnesia startup was very brain damaged -- fixing up the local data from disc, then throwing it all away to load from peers is a lot of wast…

- One big issue is that when there is a replication stream going between peers it will bottleneck how many fragments can be updated.

- Large tables make restarts really slow (have to read everything from ETS).

- Table dumps cause spikes in load.

- Uses a lot of RAM.

For net partitions, I just alert on it and fix it manually. Mnesia isn't bad as a cache, bad as a database.

Re: Erlang/OTP by Example

#75
post #70

Earlier quoted context omitted.

I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.

Bursting is an interesting thought. I think, if you planned it out, it could be done -- subject to some constraints. It would be hard to burst stateful (mnesia) nodes --- schema operations require a lock across all the nodes in the schema, and that lock requires that the nodes not be in the middle of the 'log dumping' process (where the global transaction log gets divided into per table logs and such), which means lo…

Our nodes are stateful. We also wrote our own dist (I didn't write it, it could be a lot better). We also have good metrics on our spikes, we just have black friday where we need to burst like crazy.

Re: Erlang/OTP by Example

#76

I'm kind of sad that Elixir is getting all the love and the cool resources since I much prefer Erlang's syntax. Thanks for this page!

I love Erlang too, but I don't think Elixir's popularity is stealing love away from Erlang. If anything, I think Elixir compliments Erlang and provides a path for people to enter the world of Erlang, BEAM, and OTP that they didn't have before. Elixir helped to lower the bar of entry. My understanding is that the Elixir team has also helped drive some improvements in Erlang, which is nice.

Personally, I love both Erlang and Elixir and hope to ride out the rest of my career on these platforms.

Re: Erlang/OTP by Example

#77

Earlier quoted context omitted.

Would you say that learning Erlang/OTP has given you transferable knowledge about building concurrent and distributed system? Did you transfer any of that knowledge to your Rust designs? Do you have an opinion about Scala/Akka?

I don't have experience with Scala/Akka. Erlang is a good language to learn clustering/concurrency, SMP, queuing theory, etc. Absolutely. If you learn about concurrency through Erlang, you'll have the right mindset . But you should also read Simon Marlow's book about concurrency in Haskell. You should learn pthreads and futexs so you can understand the building blocks of mailboxs and channels and higher level pattern…

So much this. I don't see myself deploying anything big in Erlang anytime soon but learning it (and building one or two small projects) gave me intuition about message passing concurrency. This knowledge transferred very nicely to Android programming with Kotlin coroutines. My colleague was astounded learning how easy it is to model concurrency with actors and channels and how much of it just works.

Re: Erlang/OTP by Example

#78

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be…

This is a post on `X` by example.

I don't think it's constructive to the discussion here to say X has drawbacks (and then go on at it), use Y (and then go on at it).

I have not used Erlang or Rust, and when I read this comment, it seemed flamewar-ish to me.

Re: Erlang/OTP by Example

#79

Earlier quoted context omitted.

One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. Also, fundamentally, I think you're way out of the norm in terms of system time to build. Getting the kinds of reliability and business value guarantees out of Rust is enormously harder than what you…

> One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. What is it you want to know? I'm giving you my personal experience, not an essay. It's hard to find people who have professional experience with Erlang, so I thought it would be useful to other peop…

> What is it you want to know?

What kind of projects have you built? What were the primary needs? What were the constraints?

> My issue is that it is enormously hard to fix the BEAM once you have built something successful with it.

There's a lot of community resources who are more than happy to give advice for free on this stuff. The fixes that usually need to be provided are easy to implement, which wouldn't be the case for a hand-rolled system. It's hard to believe you hit an actually-unique problem unless it was pre-2015, and the world of today's BEAM is an entirely different place than before it, which should itself be considered.

The dismissal of Elixir I think says a lot about your view of the ecosystem- more time and energy has been put into that half of the community in the past few years than Erlang had in the previous decade(which says a lot about just how good Erlang was before the Elixir community came along).

> We also know that people are adopting highly reliable and scaleable systems. It's just not Erlang. Cloud enables this. Containerization enables this.

This really seems like a complete misunderstanding of the kinds of easy bonuses and guarantees you get in the ecosystem. Implementing the kinds of things you're describing literally requires teams- teams! - of people to build and run. My employer is currently in the rampup to this, and the amount of time, energy, and rough edges in the ecosystem right now is out of control. Containerization is great, cloud is great, but they don't automatically give you hot deploys, easy data handoff, automatic introspection, and opposingly, they're difficult to build and debug(since the tools for such are designed for a very different Unix world), have poor separation of concerns(half the tools that handle this stuff also do 2 or 3 other things, all in different ways, and all with very blurry boundaries), and I really don't think finding knowledgable people for them is much better considering half the tools have existed for less than 4 years(coincidentally the same length of time as Erlang's newest comer).

> Instead of Erlang's IPC, you can use an IDL and an RPC compiler.

To me, this sounds like saying, "Instead of this ultra-fast and agile big rig, you can put together a raft with these here twine". Once again- you've gotta build the world, you lose the niceties of the ecosystem, and it's just time time time.

That's not to say that an RPC can't be excellent- it absolutely can- but it isn't easy without a lot of infrastructure.

> Instead of pids, you can have a service registry using etcd or DNS. You don't have to hand roll any of that.

Who uses pids anymore when you've got `bitwalker/libcluster` providing service discovery via whatever mechanism you want(including etcd, Kube DNS, Kube selectors, Consul, EC2 tags...)?

> What Rust gives you is the ability to quickly build correct, fast, and maintainable systems that are easy to scale because concurrency is a central theme of the language.

While I definitely believe that(Rust is nothing if not a real marvel of engineering), it's got a long way to go before it starts removing any reason to use the BEAM. You call those systems maintainable- but how long have you been maintaining them? What team sizes do you often work in, do your projects require? BEAM systems are famous for scaling to tens or hundreds of millions of concurrent users, with no downtime, on engineering teams that could share a couple of pizzas.

I think Rust will get there- and that's the whole reason I've spent so much time in it- but we're just barely starting to get quality Actor system implementations, and they haven't been made easy to use either.

> What languages have you learned? Did you learn about manual memory management in school/camp/on your own? What's been difficult about it for you? I'll admit, I know a lot of languages. Most of the things in Rust are familiar to me outside of lifetimes, which I don't consider to be to difficult to learn. It's just making something you consider implicitly explicit.

I didn't learn about manual memory management until very late(~3 years ago), so that's definitely where a lot of the introductory difficulty has been. But in addition, the depth and complexity of the type system, how that type system interacts with its memory model, the inconsistencies of different types because of pre-implemented traits, etc. Simply reading the Rust book took well over a month of serious study- learning Elixir via "Programming Elixir"(which I consumed before learning Erlang) taught me the majority of the language in an afternoon, and the basics of OTP by the end of the book(later that week of light reading).

Now, years later, I wouldn't consider OTP that difficult to learn or understand. But this one I'll forfeit, if only because I seem to have understood it naturally a bit faster than most(because for me, the concepts honestly seemed to "just make sense". I remember several times thinking, "This is exactly how I would build this.", which actually allowed me to forget a lot that I learned to understand and effectively program in other languages, especially around concurrency).

> We have hired three new grads who we started on Rust, and we got them up in running in a few weeks. They did have the books, mentorship, and assigned work involving small tasks to ramp up on.

Admittedly, I've had several things blocking me on this:

1. The Rust Book, while good, was long and didn't always use the best examples.

2. I only have one serious Rustacean that I can access regularly(a former Mozilla employee).

3. I've struggled to find projects that made me genuinely think, "Rust would be perfect for this", outside of small callouts to it from other languages. I'm just now on one that is going to have some unusual math that needs to be performed quickly and continuously that I might use it for, but even still.

Because of this, I'd definitely concede that it might be possible to do it much faster if one had adequate support, but without prior memory management and deep type systems experience, I doubt it'd be sub 2 months without a constant pair(which might do the trick, and I advocate in most regards anyhow).

I apologize, reading over this I see many places this likely comes off as hostile, and that's not my intention. Just a lot of your comments honestly surprise me, and directly contradict both my lived experience with the ecosystem and that of teams I hold a lot of respect for.

Re: Erlang/OTP by Example

#80

Earlier quoted context omitted.

I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -spe…

Erlang's syntax can be learned in a few hours. The language is very small. How is that a "complex learning curve?"

Erlang the language is a small part of Erlang the platform. Erlang the language is simple; Erlang + OTP + BEAM VM, which is what Erlang actually is, less so.
Post reply on HN