Live data from Hacker News

Tell HN: I'm writing an Erlang recipes book, are you interested?

leanpub.com

31–40 of 42 posts

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#31
post #29

Earlier quoted context omitted.

You have to make the producer synchronous to the consumer so the consumer is slowed down. Ask for an acknowledgement on reception of any task/event. This will instantly limit the rate at which data can come in the final worker.

Well, that breaks the whole philosophy of async message passing in Erlang, where sending message always succeeds and is not dependent on the receiver. What you are suggesting is in fact limit the queue to be of size of 1 (if only one producer, or to the number of producers). This is not what queue is all about

No, this doesn't break the async philosophy. The synchronous sequence of calls is done by a question/reply made out of 2 asynchronous calls. It's just that simple to make something synchronous, as opposed to turning something synchronous asynchronous (by using a queue).

By doing this you are not limiting the queue to the size of 1. If you have N producers, you can have more than N elements in the queue, but the idea is that each producer can only enter them one by one, with the agreement of the consumer. Nothing would stop the consumer from accepting a thousand of them before waiting before replying, therefore slowing down the producer. The idea is just to limit the rate at which you accept queries by tying it to the speed of the consumer.

If this is not enough, you can look at things like the jobs tool (https://github.com/esl/jobs) to work with scheduling.

By the way, you're mistaken on the idea of message sending always succeeding. The idea is that you will never error out because of trying to send a message (send and pray), not that the message will make it to destination. If you want any guarantee that the message made it, then you need that acknowledgement async message being sent back to you.

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#33

For reference I tried to put a range of $ into your form (30 - 50), and it gave me an unfriendly error and cleared all of the fields. You might throw some client-side validation on those fields to help rebels like me who can't follow simple directions. =)

Good point. I'll take care of it.

Thanks for letting us know!

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#34
post #29

Earlier quoted context omitted.

You have to make the producer synchronous to the consumer so the consumer is slowed down. Ask for an acknowledgement on reception of any task/event. This will instantly limit the rate at which data can come in the final worker.

Well, that breaks the whole philosophy of async message passing in Erlang, where sending message always succeeds and is not dependent on the receiver. What you are suggesting is in fact limit the queue to be of size of 1 (if only one producer, or to the number of producers). This is not what queue is all about

If you want a message queue bounded to length N, then make the messaging protocol such that the sending process sends up to N messages without an ack.

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#35
post #28
post #20

Erlang recipes, not that much. Make that Erlang/OTP recipes with a strong emphasis on the OTP side of it, and a million times yes ! Or to say it slightly differently : I'm not that much interested in recipes about the Erlang language itself, but I'm very very interested in recipes/examples of architectures of actual Erlang/OTP softwares. I'm doing mostly Python these days, but I worked on a couple of side projects in…

Have you taken a look at "Erlang and OTP in Action" ( http://www.manning.com/logan/ ) ?

Thanks for the link, this book seems to be what I was looking for !

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#36
post #29

Earlier quoted context omitted.

Well, that breaks the whole philosophy of async message passing in Erlang, where sending message always succeeds and is not dependent on the receiver. What you are suggesting is in fact limit the queue to be of size of 1 (if only one producer, or to the number of producers). This is not what queue is all about

If you want a message queue bounded to length N, then make the messaging protocol such that the sending process sends up to N messages without an ack.

The point I am trying to make is that this fundamental feature of message queues (limit their length to prevent the memory exploding) should be provided by Erlang and not reimplemented over and over again by developers.This is really a basic feature in any message queue

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#37
post #36

Earlier quoted context omitted.

If you want a message queue bounded to length N, then make the messaging protocol such that the sending process sends up to N messages without an ack.

The point I am trying to make is that this fundamental feature of message queues (limit their length to prevent the memory exploding) should be provided by Erlang and not reimplemented over and over again by developers.This is really a basic feature in any message queue

Why is this a fundamental feature of message queues any more than of other things?

(you can make a data structure which makes memory explode, for instance lists:seq(1, 123456789123456), you can make a program which makes memory explode, for instance by growing the stack enough. Why is a message queue special?)

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#38

Yes! Especially if you can cover mutual supervision - that sort of arrangement seems to get glossed over in the "tree of supervisors" world.

I'm intrigued by the idea of mutual supervision. This implies that both process dynamically become the supervisor of each other. By default, OTP supervisors are more architectural in nature -- they will always give you a top-down tree rather than a cyclic graph. Who restarts who? That's a bit tougher to do when everything becomes cyclic. Links are bidirectional though, so when one of the workers die, the other can kn…

doesn't heart kinda solve the cyclic "who restarts who thing". I think heart (if enabled) sits there until the erlang process exits, restarts the erlang vm, which in turn also brings up an instance of heart. yeah, complicated and not perfect, but it works.

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#39
post #35
post #28

Earlier quoted context omitted.

Have you taken a look at "Erlang and OTP in Action" ( http://www.manning.com/logan/ ) ?

Thanks for the link, this book seems to be what I was looking for !

I got it and I like it quite a bit. It is a good introduction book for practical Erlang use, since you learn to do things the "OTP way" right off the bat.

It has examples on how to build network servers (TCP RPC server, HTTP server, a caching service, setting up a distributed system with multiple nodes, resource discovery, logging, release managing, interfacing with C code and many other useful things).

Re: Tell HN: I'm writing an Erlang recipes book, are you interested?

#40
post #38

Earlier quoted context omitted.

I'm intrigued by the idea of mutual supervision. This implies that both process dynamically become the supervisor of each other. By default, OTP supervisors are more architectural in nature -- they will always give you a top-down tree rather than a cyclic graph. Who restarts who? That's a bit tougher to do when everything becomes cyclic. Links are bidirectional though, so when one of the workers die, the other can kn…

doesn't heart kinda solve the cyclic "who restarts who thing". I think heart (if enabled) sits there until the erlang process exits, restarts the erlang vm, which in turn also brings up an instance of heart. yeah, complicated and not perfect, but it works.

Heart does solve it in a way, yes. Any Master/Slave pattern where a masterless slave becomes a master also solves it. The complex cases I had in mind happen when you dynamically add more workers and try to set up crazier links (A is the master of B, B is the master of C, C is the master of A), although there are few practical uses of that. You'd have to try to shoot yourself in the foot I guess. I see it as a bit hard to provide a generic OTP pattern to do that kind of thing past the standard takeover/failover setup (and heart for VMs)
Post reply on HN