Ruby and Elixir: Polyglottin' FTW • Hacker News Board

Re: Ruby and Elixir: Polyglottin' FTW

#11

Post by dagw » Fri, Mar 06, 2015, 4:30 PM UTC

pjmlp wrote: post #10

themartorana wrote: post #4

Best thing I ever learned - probably 15 years ago at this point - is just use the right tool for the job. We're a mobile game company, and we use C, C#, Objective C, Java, Python, Go, Erlang, JavaScript, Lua, and more where they make sense. I love language talk and debates (and hate the language hate) here on HN. But if you're only using a single language in production, you're likely not that big, or more likely have…

I never got those "I am language X developer". It is like "I only do hammers and nails. For drilling ask the guy over there."

On the other hand if I want some intricate decorative wood carving, I'm going to seek out a guy who specializes in intricate decorative wood carving, not a guy who says "well I've never done any carving, but I've sawed, drilled and hammered lot of wood in my day and carving is basically the same thing."

Re: Ruby and Elixir: Polyglottin' FTW

#12

Post by pjmlp » Fri, Mar 06, 2015, 4:49 PM UTC

dagw wrote: post #11

pjmlp wrote: post #10

Earlier quoted context omitted.

I never got those "I am language X developer". It is like "I only do hammers and nails. For drilling ask the guy over there."

On the other hand if I want some intricate decorative wood carving, I'm going to seek out a guy who specializes in intricate decorative wood carving, not a guy who says "well I've never done any carving, but I've sawed, drilled and hammered lot of wood in my day and carving is basically the same thing."

From my point of view you are mixing tools with domains.

Decorative wood carving still requires multiple tools and is akin with something like e.g. distributed systems, not language/ product X.

Re: Ruby and Elixir: Polyglottin' FTW

#13

Post by sanderjd » Fri, Mar 06, 2015, 4:58 PM UTC

ankurpatel wrote: post #7

Why are you adding an extra level of abstraction in between Rails and Sidekiq? Why not just have Rails push to redis queue and elixir process finishes the job of sending email or doing what it has to do. Why have Sidekiq run at all? It seems adding multiple layers as such will make it harder to understand, debug and maintain in future. EDIT: Also your example of polling on Redis queue using elixir is very inefficient…

I think it makes some sense to use the Sidekiq client, because it does a lot of the tedious work of getting data into redis in a well-defined and battle-tested fashion. But certainly it doesn't make sense to run the Sidekiq daemon in this configuration.

Re: Ruby and Elixir: Polyglottin' FTW

#14

Post by sanderjd » Fri, Mar 06, 2015, 5:01 PM UTC

pmontra wrote: post #8

dagw wrote: post #3

Earlier quoted context omitted.

While Ruby and Elixir have quite similar syntax, their underlying run time models are fundamentally very different.

Yes very different model, and you realize that the syntax is not so similar once you really start working with Elixir: you start using constructs that don't belong to procedural languages at all. Similarities don't go much further than def do end (but there are too many do and too many different def in Elixir) and the sensible naming of some Library.functions() to match classes and methods in Ruby's standard library.…

Yeah, I wonder if Elixir's superficial similarity to Ruby was actually a mistake. It makes some people think they can just write Ruby and have it run faster, only to become frustrated, and others scoff at it due to prejudice against Ruby. On the other hand, I think it has a really nice syntax that works very well for its semantics, so from a purely technical standpoint it was a good choice.

Re: Ruby and Elixir: Polyglottin' FTW

#15

Post by jghn » Fri, Mar 06, 2015, 5:07 PM UTC

dagw wrote: post #11

pjmlp wrote: post #10

Earlier quoted context omitted.

I never got those "I am language X developer". It is like "I only do hammers and nails. For drilling ask the guy over there."

On the other hand if I want some intricate decorative wood carving, I'm going to seek out a guy who specializes in intricate decorative wood carving, not a guy who says "well I've never done any carving, but I've sawed, drilled and hammered lot of wood in my day and carving is basically the same thing."

Part of the problem is that the set of people who need intricate decorative wood carving and the set of people who think they need intricate decorative wood carving are very different. In fact, one is much larger than the other.

Re: Ruby and Elixir: Polyglottin' FTW

#16

Post by mmcclure » Fri, Mar 06, 2015, 5:41 PM UTC

We've done similar things at work (using a Redis queue and a Go co-process), but I actually really like the idea of using a fairly common and well-liked delayed job interface in Rails and swapping out the worker underneath. Cool idea.

One minor suggestion for the author (post doesn't have a comment section). `:timer.sleep(10)` is explained as "sleep for 10 seconds", but sleep takes millis as an argument (http://www.erlang.org/doc/man/timer.html#sleep-1).

Re: Ruby and Elixir: Polyglottin' FTW

#17

Post by Dirlewanger » Fri, Mar 06, 2015, 5:49 PM UTC

Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?

Re: Ruby and Elixir: Polyglottin' FTW

#18

Post by ankurpatel » Fri, Mar 06, 2015, 6:12 PM UTC

wcummings wrote: post #9

ankurpatel wrote: post #7

Why are you adding an extra level of abstraction in between Rails and Sidekiq? Why not just have Rails push to redis queue and elixir process finishes the job of sending email or doing what it has to do. Why have Sidekiq run at all? It seems adding multiple layers as such will make it harder to understand, debug and maintain in future. EDIT: Also your example of polling on Redis queue using elixir is very inefficient…

Redis has blocking list primitives which you can use to build an efficient queue (push and pop in O(1)), if you look at the code you'll see he's using brpop

I agree but there is no need to poll. Polling is a poor mans solution in a world where you can have persistent connection and get push notification of when a new item is added to the queue and there is no need to throttle for 10 seconds before polling again.

Re: Ruby and Elixir: Polyglottin' FTW

#19

Post by phamilton » Fri, Mar 06, 2015, 6:14 PM UTC

Dirlewanger wrote: post #17

Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?

Since Ruby's GIL means you can't do true parallel processing, it's common to kick off multiple worker processes and have them each take one job at a time.

If the job involves an expensive API call it will tie up a worker for the duration of the call.

So it's not so much ~thousands of jobs per second as it is a limitation of only 10 ruby processes per 1GB of memory. (A ruby process with the Rails env loaded is about 100MB.)

Elixir gives you the ability to make a few orders of magnitude more calls in a tiny amount of memory. (100,000 Elixir "workers" can be had in less than 1GB).

Re: Ruby and Elixir: Polyglottin' FTW

#20

Post by mperham » Fri, Mar 06, 2015, 6:22 PM UTC

Dirlewanger wrote: post #17

Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?

Sidekiq runs 25 worker threads by default. MRI will context switch upon I/O so typical server-side applications will get plenty of concurrency, even with the GIL, because they spend lots of time talking to the DB, memcached, redis, 3rd party APIs, etc.

tl;dr - Sidekiq can do a LOT of work, even on MRI with the GIL.