Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

Post reply on HN
Print view View original

231–240 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#231

Post by tanin »

lgessler wrote: post #179
tanin wrote: post #166

Earlier quoted context omitted.

Is Spec simply a test? Or is it something more? I keep hearing about it, but it seems to be more than just a test case. Let's take a simple example. I have an object with the method named "get". But I call "fetch" in my code. When will I see this error? During compile time or run time?

Think of it this way: spec gives you an easy way to ensure that your data conforms to arbitrary predicate functions. Suppose you have an XML format, that has books and authors[1]. The elements look something like this: , . Obviously, you want to make sure that book's author-id attribute will always refer to an author that actually exists. This is something a (Java-style) static type system can't do: it doesn't know a…

Thank you. I think I have a good idea what it is now. It's a constraint of a field. But it goes a bit further to ensures that, for example, an author id refers to a valid entity.

Re: Clojure at Netflix (2013) [slides]

#232

Post by gleenn »

kbp wrote: post #126
diab0lic wrote: post #115

Earlier quoted context omitted.

And now that I've responded to the GP, to address your points: > that means clojure is still being used for new projects, or not? This has been / will always be a professional choice of the engineer(s) starting a new project at Netflix. Clojure is great for a lot of reasons and lets you target JVM/NodeJS at the same time (our two largest backend languages) but as a LISP most people aren't going to be excited about us…

> but as a LISP most people aren't going to be excited about using it. I agree that Clojure has a lot of strong points and that s-expressions probably put a lot of people off, but as a Lisp programmer, I was very disappointed in Clojure's debugging/interactive development story (and I've heard that from a lot of others). It feels more like using a typical scripting language compared to the traditional Lisp/Smalltalk…

The latest Clojure in the pipe, 1.10, is supposed to have a lot of work done fixing stacktraces FWIW.

Re: Clojure at Netflix (2013) [slides]

#233

Post by dnautics »

lostcolony wrote: post #124
dnautics wrote: post #107

Earlier quoted context omitted.

What are the performance issues? People (e.g. Instagram) deploy scaled apps in Python, which is far less performant. Bleacher report obsesses about information push latency, and they use an all elixir stack that deploys to at least a million users. Not to mention WhatsApp.

CPU bound loads won't benefit from Elixir. If most of what you're doing is a little minimal data transformation, but otherwise shuffling data to different network locations, and you want consistent latency/throughput, Elixir is a dream. If you're doing some tight number crunching, then not so much.

Exactly. Who in sv is actually doing tight number crunching all the time? I think most of what we do is shuffling data around. If you need it, you can always also just orchestrate a container with a computational task anyways. And then someone's gotta marshal the data and metadata to that container... Might as well be something dreamy.

Re: Clojure at Netflix (2013) [slides]

#234

Post by iLemming »

TheMagicHorsey wrote: post #43

I love Lisp, so I thought Clojure would be a great productivity booster. And it is, if you are a one-man or one-woman shop. But try to build a team around a Clojure project and it's another story. The language basically begs you to make "magic" happen with domain-specific constructs. That might make you feel powerful as a programmer, but it's also a nightmare for new team members to get up to speed on. And then, as o…

It's a known fact that it's possible to build and maintain serious projects having only a few Clojure developers. And either because of that or for some other reason, it's been decided that it's hard to build large codebases in Clojure. That's not true though. I personally worked in a large, distributed team on quite large Clojure projects.

Re: Clojure at Netflix (2013) [slides]

#235

Post by Jach »

ken wrote: post #162
lgas wrote: post #84

Earlier quoted context omitted.

Having spent the last two years working professionally in Haskell after having spent the previous two years working professionally in Clojure, I'd disagree with this. Haskell is roughly infinitely better than Clojure for meaningful refactoring. Clojure gives you (great) tools for using your own brain to make sure the refactor goes well. Haskell replaces your brain almost entirely in the process and just gives you a p…

Just as with the word "developer" [1], I'm beginning to think that the word "refactor" is a dirty word. Are we even talking about the same activity? How is it people are alternately claiming that Haskell and Clojure (virtual opposites on the language spectrum) are infinitely superior to each other at the same thing? Wikipedia defines it as "the process of restructuring existing computer code without changing its exte…

"Refactoring" has a definitive origin, which is Martin Fowler's Refactoring book from 1999.

Fowler wanted to write the book using Smalltalk, but because the techniques he wanted to write about were fairly language agnostic, he did it in Java, as that was more popular. Unfortunately I think a lot of programmers missed the point, and now think refactoring is only something that can be done well in languages like Java (static typing) and with IDE support.

He's announced a second edition[0] in which he'll use JavaScript, to again prove the point that the techniques matter less than the language (other than sometimes techniques that work for class-based designs aren't as relevant as for function-based designs) and because JS is so popular. I'm not optimistic it will help anyone see the underlying point, but at the very least it might kill the idea that refactoring has to be hard in non-static-typed languages.

[0] https://martinfowler.com/articles/201803-refactoring-2nd-ed....

Re: Clojure at Netflix (2013) [slides]

#236

Post by diab0lic »

alextheparrot wrote: post #98
diab0lic wrote: post #63

Earlier quoted context omitted.

I am very glad you asked! I wrote and deployed (to production) some Clojure code at Netflix just yesterday. Among other things at Netflix the Mantis Query Language (MQL an SQL for streaming data) which ferries around approximately 2 trillion events every day for operational analysis (SPS alerting, quality of experience metrics, debugging production, etc) is written entirely in Clojure. This runs in nearly every criti…

Do you know offhand how MQL compares to the Apache Calcite [0] extension of SQL that some big data platforms are using for streaming SQL? [0] https://calcite.apache.org/docs/stream.html

I came back to add: One of the things that just popped into my mind that Calcite has that we don't is query planning. Obviously since MQL was developed in house we didn't get any optimizations out of the box. We've since added a layer that processes the parse trees looking for optimizations and while it isn't quite query planning it has yielded some significant performance gains (or rather poor performance mitigations, haha).

I'd imagine getting this for free from Calcite would have been really nice.

Re: Clojure at Netflix (2013) [slides]

#237

Post by nybble41 »

PopsiclePete wrote: post #197
nybble41 wrote: post #194

Earlier quoted context omitted.

I don't think laziness is really that much harder to reason about, unless you're trying to fit it into a mental model based on eager evaluation. Evaluation occurs when the result is needed; that's so hard about that? And it's not really about making algorithms look "elegant" but rather about composiblity and reuse: it's much easier to add eager evaluation to non-strict interfaces than it is to get non-strict evaluati…

What makes lazyness difficult for me isn't the mental model, it's the fact that when you get an error, the error is far from its call-site, and the truly helpful information that I could've had in the stack trace is gone. I'm looking at the error - but which lazy operation in the sequence caused it? Where? When? I by no means claim to be a Clojure guru, and I'm sure people who have more experience can just look at it…

> when you get an error, the error is far from its call-site, and the truly helpful information that I could've had in the stack trace is gone

I'll grant you that one. Languages with non-strict evaluation (or tail calls, or inlining, or extensive use of macros) rarely have useful stack traces. You can see what was being evaluated but not necessarily how it got there. The ability to leverage the stack as a debugging aid is a feature of certain runtime conventions more applicable to imperative programming paradigms than to functional ones. In modern Haskell there is a way for functions to receive extra information about their call site as an implicit parameter, which helps with error reporting. I'm not sure what tools Clojure might offer along these lines.

I will point out that tracing the source of incorrect or corrupted data is no less of a problem for programs written in strict, imperative languages. Functional programming adds to this by emphasising functions as a form of data, and non-strict evaluation carries that still further by treating all data as closures. On the other hand, strong static typing limits the kinds of runtime errors which lay hidden within those closures to a manageable set (assuming a pure functional language like Haskell).

>> Either way, making eager evaluation the default is a form of premature optimization.

> I don't see how that's true - it's the default way CPU's operate on the silicon level.

That's not really true, given the existence of speculative and out-of-order execution. In any case, how the CPU operates at the silicon level has very little to do with whether a high-level language optimization is premature.

> Millions upon millions of lines of non-lazy code have been written and put into production and they perform real, valuable work.

I'm not disputing that, but it's beside the point. Code which has been prematurely optimized can still "perform real, valuable work". It just may take a bit longer to run or require more resources to develop (for example by being harder to reuse, requiring more code to be written to accomplish the same result).

Re: Clojure at Netflix (2013) [slides]

#238

Post by jwhitlark »

justinhj wrote: post #171
m12k wrote: post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

Scala is sometimes referred to as the Haskellator, in that once you get far enough down the pure functional programming road, you may find yourself wanting to switch to Haskell. Scala doesn't enforce pure functional programming as much as Haskell. On the other hand I believe that Scala is much more widespread in industry because it's easy to sneak it in the door as an OOP/imperative language that just happens to have…

After my time with Scala, I think that if I need a heavy type system, I'd look to Haskell. And there are certainly classes of problems that I would choose a type system, but the general case isn't nearly as strong as I though it was before seeing them used in anger.

So, in the following, I'm not ranting at you, but I am saying I see lots of arguments for types that jump to QED without nearly enough evidence, with the "everybody knows" kinda explanation. And you happen to have put a handy list right in front of me when I had time to respond.

> ... difficulties with refactoring, runtime errors, needing lots of test coverage and not scaling well to large teams and codebases.

I was really impressed with Scala when learning or toy projects, but every time I saw it in production it was pretty horrid. Small, technical refactoring was taken care of by the IDE, but large scale refactoring takes the same amount of thought as before, but now there's 10x as much code to understand first.

There were less runtime errors, but not much less, and I saw a lot of code that didn't error, but didn't get done what it needed to.

When it comes to scaling to large teams, I saw lots of "The compiler says it's ok, and the tests pass, so commit!" where the code didn't make sense! The old thing about write code for programmers to understand first, and the machine to understand second...

Outside of certain specific situations, I just haven't seen the benefits outweigh the increased code weight and slower time to market.

Re: Clojure at Netflix (2013) [slides]

#239

Post by lostcolony »

jb3689 wrote: post #95
udfalkso wrote: post #88

Earlier quoted context omitted.

Another option worth considering is Elixir. It's also dynamically typed, but the pattern matching helps mitigate dynamic type issues if used correctly. It's a pleasure to work with.

I love Elixir but it's worth noting that I'd be hard-pressed to recommend it for every project because of the performance and base OS integration characteristics of the BEAM. The BEAM is a truly amazing piece of software and I do think that Erlang and Elixir are the true microservices dream we all wanted, but the BEAM uses preemptive scheduling and doing things like spinning up a new OS process can be a lot more work…

I'll also point out, I'd much prefer a default of Elixir, than of Java.

Why? Because most of my problems are things Elixir can solve well, and Java can solve moderately well, after a bunch of pain and tweaking. Default to the thing that can solve most of your problems well, and pick a different tool when needed, rather than default to the thing that can solve all of your problems moderately well, and you never pick anything else, and so never have any elegant, quickly built yet well designed, solutions.

Re: Clojure at Netflix (2013) [slides]

#240

Post by blandflakes »

xapata wrote: post #140
jerf wrote: post #123

Earlier quoted context omitted.

Haskell does REPL-driven development fairly well. It's not the best REPL, but it works, especially if you take a moment to learn a few of the commands to drive it. Dynamic types are not necessary for REPL-driven development.

Yes, that's why becoming happy with REPL-driven development is more important than becoming happy with dynamic typing. I think much of the productivity boost attributed to dynamic typing is in fact caused by heavy use of a REPL, and not dynamic typing.

I'd maybe consider that perhaps becoming happy with REPL-driven development isn't a universal solution. It's not like I didn't understand the concepts, the tools, or the process. I just didn't find that it made me any more proficient at building software.
Post reply on HN