Building Clojure Services at Scale • Hacker News Board

Re: Building Clojure Services at Scale

#11

Post by arohner » Fri, Sep 13, 2013, 5:51 PM UTC

One thing I've noticed with my own Clojure service at scale, is that eventually, I want to significantly reduce my use of futures entirely.

When you're getting started with Clojure, it's easy to do everything that needs to happen "later" or "in the background" as a future. Once you go to production, you realize that about half of those futures should have been jobs going into a queue somewhere, with logging and guaranteed delivery, retrying on another machine, if necessary.

I haven't gotten to the point of writing that library, but I probably will, soon.

Re: Building Clojure Services at Scale

#12

Post by prospero » Fri, Sep 13, 2013, 6:16 PM UTC

arohner wrote: post #11

One thing I've noticed with my own Clojure service at scale, is that eventually, I want to significantly reduce my use of futures entirely. When you're getting started with Clojure, it's easy to do everything that needs to happen "later" or "in the background" as a future. Once you go to production, you realize that about half of those futures should have been jobs going into a queue somewhere, with logging and guara…

Hey Allen, at Factual we're working on something in that vein: https://github.com/factual/skuld. As you might expect, doing it right takes something a bit more ambitious than just a library.

If you're interested in collaborating, or just have thoughts on how it could be more useful for you, let me know.

Re: Building Clojure Services at Scale

#13

Post by yummyfajitas » Fri, Sep 13, 2013, 6:29 PM UTC

kodablah wrote: post #8

jhspaybar wrote: post #3

This is the first I've heard of the "Scala runtime". Isn't it compiled to bytecode and usable as a jar dropped in anywhere on the JVM possibly with some name mangling if you don't design your public interface to be pretty for java based projects?

This is becoming more and more common for Scala projects that want to build libs in Scala and only export a Java interop API (as opposed to pure Java at the bottom and building Scala on top). E.g. https://github.com/apache/kafka/tree/0.8/core/src/main/scala...

The real difficulty comes when you want to use multiple scala libraries in a project which are compiled against different scala runtime versions.

I live in scala land, but I prefer java libraries for this reason.

Re: Building Clojure Services at Scale

#14

Post by gtani » Fri, Sep 13, 2013, 7:04 PM UTC

jhspaybar wrote: post #3

This is the first I've heard of the "Scala runtime". Isn't it compiled to bytecode and usable as a jar dropped in anywhere on the JVM possibly with some name mangling if you don't design your public interface to be pretty for java based projects?

If you want writeups on scala/java interop, the Manning "in Action" and "in Depth" books have excellent summaries. From scala you need to worry about java annotations, static members and "throws" exception declarations. Java private /protected etc are checked at runtime and compile time. (That's just me copying items out of the "In Action" ToC, but everything's very well documented).

For clojure/java interop, there's also good writeups in 2 Manning books, and also the OReilly and Pragmatic books. It looks like combining them (Scala/clojure interop) would take some plumbing, depending on the specific libs involved but i havent' tried. (And the Manning authors are all pretty far along on 2nd editions of both "Joy" and "In Action")

http://www.manning.com/fogus2/

Re: Building Clojure Services at Scale

#15

Post by adambard » Fri, Sep 13, 2013, 8:13 PM UTC

arohner wrote: post #11

One thing I've noticed with my own Clojure service at scale, is that eventually, I want to significantly reduce my use of futures entirely. When you're getting started with Clojure, it's easy to do everything that needs to happen "later" or "in the background" as a future. Once you go to production, you realize that about half of those futures should have been jobs going into a queue somewhere, with logging and guara…

I've had great success using Lamina for in-process message queues in lieu of futures, although this does nothing towards making your system distributed.

Re: Building Clojure Services at Scale

#16

Post by cgag » Fri, Sep 13, 2013, 8:29 PM UTC

adambard wrote: post #15

arohner wrote: post #11

One thing I've noticed with my own Clojure service at scale, is that eventually, I want to significantly reduce my use of futures entirely. When you're getting started with Clojure, it's easy to do everything that needs to happen "later" or "in the background" as a future. Once you go to production, you realize that about half of those futures should have been jobs going into a queue somewhere, with logging and guara…

I've had great success using Lamina for in-process message queues in lieu of futures, although this does nothing towards making your system distributed.

In process queues are a good abstraction, but that doesn't really protect you in case of your app crashing, which I think is the concern more than being distributed. I may be misunderstanding though.

Edit: I was already a big fan of SoundCloud, very pleased to find out they use Clojure.

Re: Building Clojure Services at Scale

#17

Post by josephwilk » Fri, Sep 13, 2013, 8:52 PM UTC

dinedal wrote: post #9

I've used Hystrix to great effect before, but am now really curious about mixing it with LMX Disruptor aka Reactor and it's clj bindings ( https://github.com/clojurewerkz/meltdown ). Anyone have any experience doing so?

That looks very interesting, will have to play!

Re: Building Clojure Services at Scale

#18

Post by prospero » Fri, Sep 13, 2013, 9:41 PM UTC

cgag wrote: post #16

adambard wrote: post #15

Earlier quoted context omitted.

I've had great success using Lamina for in-process message queues in lieu of futures, although this does nothing towards making your system distributed.

In process queues are a good abstraction, but that doesn't really protect you in case of your app crashing, which I think is the concern more than being distributed. I may be misunderstanding though. Edit: I was already a big fan of SoundCloud, very pleased to find out they use Clojure.

I think the point was that it's hard to make something failure-tolerant without being distributed. A locally persisted queue would avoid certain failure modes, though.

Re: Building Clojure Services at Scale

#19

Post by jamii » Fri, Sep 13, 2013, 9:57 PM UTC

dinedal wrote: post #9

I've used Hystrix to great effect before, but am now really curious about mixing it with LMX Disruptor aka Reactor and it's clj bindings ( https://github.com/clojurewerkz/meltdown ). Anyone have any experience doing so?

I was recently looking for clojure bindings. Meltdown is the only one I could find that is currently maintained but it doesn't appear to expose consumer barriers anywhere. Without them the disruptor is just a lockless ring-buffer. The barriers are what allow building multi-threaded pipelines on top of it.

Re: Building Clojure Services at Scale

#20

Post by josephwilk » Fri, Sep 13, 2013, 10:14 PM UTC

jeremyjh wrote: post #4

jhspaybar wrote: post #3

This is the first I've heard of the "Scala runtime". Isn't it compiled to bytecode and usable as a jar dropped in anywhere on the JVM possibly with some name mangling if you don't design your public interface to be pretty for java based projects?

That's right, its really just a library. Also, Finagle already has an idiomatic Java API so using it from Clojure would not require use of any mangled names.

Thanks for those replies. I've updated that section to better reflect Scala just being a library dependency.

I'm looking forward to playing with Finagles Java API as at some point I'm going to need to connect our Finagle projects with our Clojure services.