Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

151–160 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#151
post #3

What happened to Clojure? Was it just a fad, or are people still using it in their day jobs? Are people still hiring Clojure devs? If you write Clojure at work, are you happy with it?

We've been using a full Clojure stack (front-end + backend) in production for nearly 4 years and very happy with it (6-8 people full time). But that is very much anecdotal. I guess one thing to note is that the language is very stable, there are very few (if any) breaking changes and a 4 year old library will "just work" most of the time. Even more amazing is that it's trivial to port 40 year old ideas from (Common)L…

"...personally I wouldn't even want a ton of extra features in Clojure ... the language is fine and libraries fill in the rest."

And for anything else there's macros.

Re: Clojure at Netflix (2013) [slides]

#152
post #61
post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

I use Clojure in the large, and the lack of static typing isn't something I miss. FWIW, spec allows you make similar guarantees if you use it (technically more, since you can express more than HM type systems). I think the static/dynamic divide speaks to deep divisions in programmer personalities, but if you're open to suggestion, I would put it like this: As Haskell/OCaML are to static langs like Java, Clojure is to…

In my experience, static typing was somewhat helpful in monolithic projects, but when I saw it used in heterogeneous distributed systems, it seemed to lose much of its benefit and impose new costs.

I'm curious if we're going to see a big new synergy between Kubernetes (represent everything as data) and Clojure (great with data).

Re: Clojure at Netflix (2013) [slides]

#153
post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

Some implementations of SML are definitely production-ready.

which implementaions of SML are the production ready ones?

Re: Clojure at Netflix (2013) [slides]

#154
post #16
post #3

What happened to Clojure? Was it just a fad, or are people still using it in their day jobs? Are people still hiring Clojure devs? If you write Clojure at work, are you happy with it?

While other posters comment that "Clojure still works", IIRC the "promise" was that Clojure/Lisp would be much, much more productive , and that claim I feel like must been disproved in daily work, otherwise Clojure would still a main topic around here. Reminds me of the classic Erann Gatt post: https://news.ycombinator.com/item?id=2308370 (I loved to play with Elisp, Clojure, Lua, Factor, Ion, Ioke and hope dynamic l…

"hope dynamic languages become fashionable again someday..."

Python, Javascript and Ruby are very dynamic.

Re: Clojure at Netflix (2013) [slides]

#155

Earlier quoted context omitted.

Hey Alex, I explored potentially using Calcite when we initiated this project. The syntax is very similar because both are an SQL dialect. Some differences are that MQL largely targets unstructured data (there is no schema for the streams it operates on - they're just streams of JSON blobs). In addition to that one of the goals of MQL was to have different compiler backends which allows different call sites to operat…

Thanks for the detailed answer - as you likely already know many projects have taken the approach you detailed in your past paragraph, it makes sense if your goal is more freedom for sure. I am wondering how you do client side data egress filtering - does each event need to get materialized, assessed for certain fields or structure, and then sent once for each outgoing stream on your sender? Seems like a good strateg…

The events are already materialized in memory as part of whatever is recording them, thankfully.

The MQL that runs in the services checks the WHERE clause of every query running and then projects a superset of all the fields necessary for every part of the query for all matching queries (including other parts that the server won't be processing: group by, order by, having, etc...). It then tags the event with all of the matching queries. This way we only need to egress a single event.

What happens next requires a lot more context about Mantis, which is a reactive stream processor. The data is egressed from the service to what is referred to as a source job in Mantis... it is the source job's responsibility to multiplex this data to any of the consuming jobs. Jobs can be subscribed to one another so this is a natural fit for Mantis.

As for scaling everything is round robin after it leaves the service and enters Mantis but obviously busier services will pay more cost because they're processing more events. Regular expressions in WHERE clauses has been a pain point for scale, and absurdly heavy queries such as `SELECT * FROM STREAM` result in data dropping as most single machine consumers can't keep up with the large streams. We have different methods of providing different delivery semantics but in the base case we're at most once and will drop data if the client can't keep up.

We have some streams that exceed 1 million RPS at certain times of the day, and last we checked the system moves around 2 trillion events per day so we've managed to scale it to the needs of the company pretty well. It has been a while since I performance tested it, but it's always been a goal to have it push as close as possible to saturating the gigabit connection on the boxes on which it runs.

Re: Clojure at Netflix (2013) [slides]

#156
post #136
post #130

Earlier quoted context omitted.

What is stopping you adding Spec to your existing code?

I’ve found that overly using spec leads to more maintenance than upside. We went all in using spec and generators when they were released and ended up having to debug the specs themselves. 2c

I think of spec like seasoning; if you put lots on everything, you ruin the dish.

Re: Clojure at Netflix (2013) [slides]

#157
post #102

amazed to see there are people who find myObject.myMethod(x, y, z); easier to read than (my-function x y z) - that's 2 delimiters in Clojure vs 6 in your C-style language

In fairness, it'd actually be: (my-method my-object x y z) But yes, the Lisp version is so much clearer to me. I just don't get why it's not more popular.

How does this work with autocomplete if you need to know the method name before the object you're working with? Maybe I'm not getting something but that sounds incredibly painful to use in practice.

Re: Clojure at Netflix (2013) [slides]

#158
post #102

Earlier quoted context omitted.

In fairness, it'd actually be: (my-method my-object x y z) But yes, the Lisp version is so much clearer to me. I just don't get why it's not more popular.

How does this work with autocomplete if you need to know the method name before the object you're working with? Maybe I'm not getting something but that sounds incredibly painful to use in practice.

Because idiomatic Clojure code is based on functions and not methods.

The method call syntax is just there to support interop with Java.

Re: Clojure at Netflix (2013) [slides]

#159

I love Lisp, so I thought Clojure would be a great productivity booster. And it is, if you are a one-man or one-woman shop. But try to build a team around a Clojure project and it's another story. The language basically begs you to make "magic" happen with domain-specific constructs. That might make you feel powerful as a programmer, but it's also a nightmare for new team members to get up to speed on. And then, as o…

"Most programmers are turned off by the syntax and/or immutability."

Programmers who don't see the value of immutability or can't understand it are not programmers you really want to work with.

Re: Clojure at Netflix (2013) [slides]

#160
post #42

Earlier quoted context omitted.

I work for one of the big "brand name" companies (I don't want to say which, but it's on the "companies using Clojure page"), and on my team we try and start all new projects to use Clojure. Frankly, I love it. After the "zen" of lisp clicked for me, Clojure gave me an OS/2-style "Better Java Than Java". Suddenly I was able to compose standard JVM library stuff in a more-natural and easy way than I could in vanilla J…

Care to mention what is the language you used in your last job that you are comparing clojure with.

I was using F#, and before that Haskell, and before that Scala.
Post reply on HN