Ruby and Elixir: Polyglottin' FTW • Hacker News Board

Re: Ruby and Elixir: Polyglottin' FTW

#21

Post by rubiquity » Fri, Mar 06, 2015, 6:38 PM UTC

I've done a setup similar to this in the past and really enjoy it. If you ever find yourself writing logic in your Resque/Sidekiq jobs that has to do with manually managing a data structure in Redis with regard to your jobs, your app is screaming for a language with better tools for modeling complex workflows.

A lot of people are hitting on the performance improvements (25 Sidekiq threads/workers versus 10k-100k+ Elixir/Erlang processes) but when I find myself doing this it's actually because my background process flows are really complex. The drastic increase in concurrency is just the cherry on top.

Background work is asynchronous by nature and often times you have dependencies between jobs, groups/batches jobs and also need robust error handling/recovery. This is where Elixir/Erlang really shines and using Actors & OTP to model these complex flows is the huge win in my experience. Sidekiq Pro offers support for batches which covers some of this, but ultimately Elixir/Erlang processes are just a huge improvement in modeling such systems.

A concrete example is sending off 10 jobs. When all 10 are done, move on to the next step. Each of these 10 jobs can also send data to the waiting step which it can then use as arguments. If any job should fail N times, cancel and rollback all of the jobs and try again. After the next step, split those 10 jobs into two groups of 5 and do those 5 sequentially. So on and so forth. Try doing this in Ruby. You'll likely have to manage your own data structures in Redis. In Elixir you just use GenServers and sometimes ets tables. It's all handled in the language itself, rather than external data stores.

Sidekiq is probably suitable for 90% of Ruby apps doing basic fire and forget and batching with Sidekiq Pro. But if your domain has seriously complicated background processing, use a tool more suited for that.

Re: Ruby and Elixir: Polyglottin' FTW

#22

Post by lshemesh » Sat, Mar 07, 2015, 1:57 AM UTC

I recently did a Sidekiq background job in ruby passing messages to a python script using zeromq. It was a blast.

Re: Ruby and Elixir: Polyglottin' FTW

#23

Post by danneu » Sat, Mar 07, 2015, 6:50 AM UTC

ankurpatel wrote: post #18

wcummings wrote: post #9

Earlier quoted context omitted.

Redis has blocking list primitives which you can use to build an efficient queue (push and pop in O(1)), if you look at the code you'll see he's using brpop

I agree but there is no need to poll. Polling is a poor mans solution in a world where you can have persistent connection and get push notification of when a new item is added to the queue and there is no need to throttle for 10 seconds before polling again.

A listening pattern still has to poll on boot for unprocessed jobs.

Re: Ruby and Elixir: Polyglottin' FTW

#24

Post by phamilton » Sat, Mar 07, 2015, 6:17 PM UTC

sanderjd wrote: post #14

pmontra wrote: post #8

Earlier quoted context omitted.

Yes very different model, and you realize that the syntax is not so similar once you really start working with Elixir: you start using constructs that don't belong to procedural languages at all. Similarities don't go much further than def do end (but there are too many do and too many different def in Elixir) and the sensible naming of some Library.functions() to match classes and methods in Ruby's standard library.…

Yeah, I wonder if Elixir's superficial similarity to Ruby was actually a mistake. It makes some people think they can just write Ruby and have it run faster, only to become frustrated, and others scoff at it due to prejudice against Ruby. On the other hand, I think it has a really nice syntax that works very well for its semantics, so from a purely technical standpoint it was a good choice.

This is an open acknowledged issue on elixir-lang-core mailing lists. Many requests to make Elixir behave like Ruby are shot down. Often, what is requested already has a powerful analog in Elixir. If not, a lot of thought goes into how to add said feature according to Elixir idioms and style rather than blindly porting from Ruby.