Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

Post reply on HN
Print view View original

131–140 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#131

Post by alextheparrot »

diab0lic wrote: post #109
alextheparrot wrote: post #98

Earlier quoted context omitted.

Do you know offhand how MQL compares to the Apache Calcite [0] extension of SQL that some big data platforms are using for streaming SQL? [0] https://calcite.apache.org/docs/stream.html

Hey Alex, I explored potentially using Calcite when we initiated this project. The syntax is very similar because both are an SQL dialect. Some differences are that MQL largely targets unstructured data (there is no schema for the streams it operates on - they're just streams of JSON blobs). In addition to that one of the goals of MQL was to have different compiler backends which allows different call sites to operat…

Thanks for the detailed answer - as you likely already know many projects have taken the approach you detailed in your past paragraph, it makes sense if your goal is more freedom for sure.

I am wondering how you do client side data egress filtering - does each event need to get materialized, assessed for certain fields or structure, and then sent once for each outgoing stream on your sender? Seems like a good strategy for reducing network bandwidth, but it might reduce your throughout moderately (Due to serialization and analysis costs) or cause hotspots (If you have a distributed stream that is partioned in a particular way)? These are normally problems each consumer individually faces, where now it compounds where the producer does that work for each of the N readers. I appreciate the idea a lot, just wondering if you’ve had any issues making it scale nicely for highly subscribed streams.

Re: Clojure at Netflix (2013) [slides]

#132

Post by jwhitlark »

lvh wrote: post #66

Latacora has its token Clojure person: me. I write Clojure every occasion I get, we've shipped Clojure code to customers, et cetera et cetera. Most of the stuff I'm doing in Clojure would be possible in other languages, but in a sense that's only one notch below "Turing tarpit" possible. So, in the only sense that a startup should care (would you f'ing ship it?): only possible in Clojure or a handful of close cousins…

I think some of that is due to the nature of the people it attracts, namely, grumpy, older, I've been burned before, programmers. I use Clojure constantly and have for 10 years, but I've never felt a strong need to proselytize. I just make money and enjoy coding, day in and day out.

Re: Clojure at Netflix (2013) [slides]

#133

Post by Rapzid »

blandflakes wrote: post #68
m12k wrote: post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

There are a lot of people who claim they're more productive in Clojure, in spite of or because of its dynamic typing. I'll just throw my anecdatum out here: I found that dynamic typing was a major pain even on my own personal projects. I also never found REPL-driven development to mesh well with my workflow. In Scala, for example, entire classes of error that I just shouldn't be able to make don't exist. Maybe I'm ju…

Never did much clojure outside Riemann "configs" but I found the experience of light table's insta-repl fantastic. Scratch pad type workflows are a fav of mine(linqpad is amazing) and insta-repl blew me away.

Re: Clojure at Netflix (2013) [slides]

#134

Post by diab0lic »

kbp wrote: post #126
diab0lic wrote: post #115

Earlier quoted context omitted.

And now that I've responded to the GP, to address your points: > that means clojure is still being used for new projects, or not? This has been / will always be a professional choice of the engineer(s) starting a new project at Netflix. Clojure is great for a lot of reasons and lets you target JVM/NodeJS at the same time (our two largest backend languages) but as a LISP most people aren't going to be excited about us…

> but as a LISP most people aren't going to be excited about using it. I agree that Clojure has a lot of strong points and that s-expressions probably put a lot of people off, but as a Lisp programmer, I was very disappointed in Clojure's debugging/interactive development story (and I've heard that from a lot of others). It feels more like using a typical scripting language compared to the traditional Lisp/Smalltalk…

Oh, yes this is another excellent example of pain while building Clojure code bases. Though I'll say I only agree with half of your statement. I've found the interactive development story to be great with nrepl/fireplace/vim but the debugging is downright terrible... this is the single biggest blocker to me using it for larger systems.

Re: Clojure at Netflix (2013) [slides]

#135

Post by mschaef »

the_af wrote: post #59
mschaef wrote: post #27

Earlier quoted context omitted.

Yes. It's still a thing... but I do think Clojure has lost mindshare. Just some impressions: * It's natural for the development of a language to slow down after an initial period of churn. So Clojure has reached a certain level of maturity where there's less 'new cool stuff'. * There are other areas of industry where there's more 'new cool stuff' and attention naturally shifts there. * The Honeymoon period is over, a…

> I know of at least one organization looking into moving away from Clojure for the purpose of having a more attractive portfolio of assets to potential outside investors. Out of curiosity, without naming the organization, can you elaborate on this? I don't understand what kind of portfolio you're talking about here, the codebase? Why would investors care whether it's done in $LANG? I thought they cared more about th…

> Why would investors care whether it's done in $LANG? I thought they cared more about the business and business-related concerns.

I think the concern is around how hard long term sustainability of the codebase. If it's built in a relatively minor tool, there's merit to the belief that it'll be hard to find people that know it well. (And this is discounting the possibility that a dynamically typed language might be less suitable than programming in the large than something that has more static checking.)

When I was 23 and fresh out of school, I'd probably called this line of reasoning ill-informed at best... after all, a good developer can learn any language, and you'd only want to hire _good_ developers anyway. Twenty years later, I'm a lot more sympathetic. If you're trying to deliver business value under time/budget/risk constraints, adding a known, short-term risk (a minority language) for an unknown, longer-term benefit is a hard sell.

I think the Lisp-like languages are particularly prone to this in that they derive so much of their power by providing lots of mechanisms for abstraction. (Higher-order functions, syntactic macros, multi-methods, etc.) If you have the right abstractions in your code it can be great, but getting to that point and maintaining it can be expensive. (And in the meantime, you also have a great deal of power for shooting yourself in the foot.)

My general sense of languages like Clojure is they are very good for people with experience and time to very carefully consider their problem space. However, that's not necessarily a great description of the modern IT profession.

Re: Clojure at Netflix (2013) [slides]

#136

Post by bm1362 »

cutler wrote: post #130
diab0lic wrote: post #104

Earlier quoted context omitted.

Hey, sorry I saw that and typed a quick response just as I woke up. I'm not usually at a computer so early in the day. I'll address these now that I'm in front of a machine. :) > Did they keep writing more Clojure? Yes but it has never been the primary language at Netflix. > How much more did they rewrite from Java to Clojure? Very little, if any was rewritten from Java. > If so, how much of their code is now in Cloj…

What is stopping you adding Spec to your existing code?

I’ve found that overly using spec leads to more maintenance than upside. We went all in using spec and generators when they were released and ended up having to debug the specs themselves. 2c

Re: Clojure at Netflix (2013) [slides]

#137

Post by diab0lic »

cutler wrote: post #130
diab0lic wrote: post #104

Earlier quoted context omitted.

Hey, sorry I saw that and typed a quick response just as I woke up. I'm not usually at a computer so early in the day. I'll address these now that I'm in front of a machine. :) > Did they keep writing more Clojure? Yes but it has never been the primary language at Netflix. > How much more did they rewrite from Java to Clojure? Very little, if any was rewritten from Java. > If so, how much of their code is now in Cloj…

What is stopping you adding Spec to your existing code?

The thing I would be getting from spec is confidence when modifying the code base, which I get from the unit tests I wrote pre-spec. If I were to initiate the project today I'd probably have less tests and more spec.

Re: Clojure at Netflix (2013) [slides]

#138

Post by jwhitlark »

ronnier wrote: post #80
projectileboy wrote: post #53

This is a bit tangential to the post, but I've seen a couple comments in this thread that amount to "Clojure is hard to read", but I think that's a really unfair comparison, because people are unwittingly comparing apples and oranges. It seems to me the reason people think Clojure is hard to read is that the language is so powerful and expressive, that when you're reading Clojure code, you're typically trying to unde…

I worked in a large company where clojure was used. After the original developers moved on from that project (they always move on), it was a giant struggle to get people who could work in the code base. They ended up rewriting it in something more standard. Whatever benifit clojure provided was outweighed by the high cost of maintaining the code base and the inability to find devs who wanted to work on it. That’s my…

I think this is a valid point. We talk about the right tool for the job, but forget that the evaluation needs to include the people using the tool, the people paying for the job to be done, and other elements of the environment.

Re: Clojure at Netflix (2013) [slides]

#139

Post by puredanger »

diab0lic wrote: post #63
codetrotter wrote: post #6

Would be interested to hear their experiences with Clojure at Netflix for the 5 years that have followed since then. Did they keep writing more Clojure? If so, how much of their code is now in Clojure compared to Java? How much more did they rewrite from Java to Clojure? Do they use Clojure rather than Java for new code? What other languages do they use? Python? Erlang? Rust? How much, in terms of functionality, is w…

I am very glad you asked! I wrote and deployed (to production) some Clojure code at Netflix just yesterday. Among other things at Netflix the Mantis Query Language (MQL an SQL for streaming data) which ferries around approximately 2 trillion events every day for operational analysis (SPS alerting, quality of experience metrics, debugging production, etc) is written entirely in Clojure. This runs in nearly every criti…

Hey, would be great to have this on our success stories page at https://clojure.org/community/success_stories. If you would be able to do that, please contact me at alex.miller@cognitect.com. Thanks!

Re: Clojure at Netflix (2013) [slides]

#140

Post by xapata »

jerf wrote: post #123
xapata wrote: post #99

Earlier quoted context omitted.

Try to grok REPL-driven development before dynamic typing. You won't like the latter until you love the former.

Haskell does REPL-driven development fairly well. It's not the best REPL, but it works, especially if you take a moment to learn a few of the commands to drive it. Dynamic types are not necessary for REPL-driven development.

Yes, that's why becoming happy with REPL-driven development is more important than becoming happy with dynamic typing. I think much of the productivity boost attributed to dynamic typing is in fact caused by heavy use of a REPL, and not dynamic typing.
Post reply on HN