Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

201–210 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#201

amazed to see there are people who find myObject.myMethod(x, y, z); easier to read than (my-function x y z) - that's 2 delimiters in Clojure vs 6 in your C-style language

TXR Lisp:

  myobject.(mymethod x y.w z.bar)
Notations can be useful. Most Lispers reach for 'expr rather than (quote expr). Rational numbers could look like #R(1 2) but CL has 1/2 instead.

I decided on a dot notation for object access because OOP is used to organize the program as a whole; so it's not some small thing like arithmetic.

The dot notation integrates into Lisp; it doesn't disturb the syntax with ambiguities and reads/prints cleanly, and corresponds to structure in several straightforward ways:

   a.b.c -> (qref a b c)
  .a.b.c -> (uref a b c)
There must not be whitespace: a . b is the consing dot, which can only appear at the end of the list. Of course, if a and b are 3 and 2, we get 3.2 which is a floating-point token; and that mustn't be glued to something else.

  1> '(qref a b c)
  a.b.c
  2> '(qref a b (qref c))
  (qref a b (qref c))
  3> '(qref a (qref b c))
  (qref a b.c)
  4> 3.2.a
  expr-4:1: trailing junk in floating-point literal: 3.2.a
  ** syntax error
  5> '(qref a (b c) d (e) (f g))
  a.(b c).d.(e).(f g)
There are qref and uref macros which do useful things with the notation. E.g.

  14> (stat "args.h")
  #S(stat dev 2049 ino 670288 mode 33204 nlink 1 uid 500 gid 500 rdev 0
          size 5628 blksize 4096 blocks 16 atime 1540848391 mtime 1524635813
          ctime 1524635813 path "args.h")
  15> (mapcar [chain stat [juxt .path .size]] (glob "*.h"))
  (("args.h" 5628) ("arith.h" 2270) ("buf.h" 3712) ("cadr.h" 2515)
   ("combi.h" 1517) ("config.h" 2644) ("debug.h" 3762) ("dict.h" 4373)
   ("eval.h" 3898) ("ffi.h" 5146) ("filter.h" 2298) ("ftw.h" 1492)
   ("gc.h" 2370) ("glob.h" 1485) ("hash.h" 3412) ("itypes.h" 3238)
   ("lib.h" 37442) ("lisplib.h" 1680) ("match.h" 2031) ("parser.h" 4597)
   ("rand.h" 1798) ("regex.h" 3204) ("signal.h" 7954) ("socket.h" 1460)
   ("stream.h" 8553) ("struct.h" 3475) ("strudel.h" 1486) ("sysif.h" 2747)
   ("syslog.h" 1998) ("termios.h" 1436) ("txr.h" 1885) ("unwind.h" 9880)
   ("utf8.h" 2590) ("vm.h" 1681) ("vmop-ORIG.h" 1660) ("vmop.h" 2008)
   ("y.tab.h" 4542))
Example OOP code that is full of the notation is found in the compiler: http://www.kylheku.com/cgit/txr/tree/share/txr/stdlib/compil...

I feel I have come up with a successful design: a way of integrating the dot selection notation into a Lisp dialect without losing "Lispiness".

Re: Clojure at Netflix (2013) [slides]

#202
post #80

Earlier quoted context omitted.

I worked in a large company where clojure was used. After the original developers moved on from that project (they always move on), it was a giant struggle to get people who could work in the code base. They ended up rewriting it in something more standard. Whatever benifit clojure provided was outweighed by the high cost of maintaining the code base and the inability to find devs who wanted to work on it. That’s my…

It saddens me to hear such stories as they show up from time to time. I can't help but feel that a lot of programmers just decide to stop learning at some arbitrary point in their career (my observations suggest that point is around the end of their university term or their first job). The industry too seems biased heavily in favor of using lowest-common-denominator, dumbest possible solutions they can get. And not "…

> I can't help but feel that a lot of programmers just decide to stop learning at some arbitrary point in their career (my observations suggest that point is around the end of their university term or their first job).

Indeed.

I had the frustrating, but ultimately fortuitous, situation of having my first job be one where I was asked to build a solution with woefully under-powered tools. I'm not even talking about the difference between hand tools and power tools - I was essentially asked to use kid toys.

That situation made me very angry, and I was constantly working overtime to make up for the lack of productivity of my tools - even then we were going to miss our deadline.

Finally, I managed to convince management to let me and my team use real tools - nothing super advanced, but well suited for the job. Very quickly we turned around the release, I started working normal hours, and when I left that job shortly after the release my employer found that they didn't need to hire a replacement because the workload was so much less.

This experience taught me that the power of your tools does matter, and that the initial pain of technology transition can easily be outweighed by the benefits of using something better.

For me, the quality I look for in my tools is: does it reward mastery? I'm not (primarily) looking the initial growth curve and how long it takes before I get to my current productivity. Rather, I'm looking at where the ceiling is.

Re: Clojure at Netflix (2013) [slides]

#203
post #183

Earlier quoted context omitted.

Hey, sorry I saw that and typed a quick response just as I woke up. I'm not usually at a computer so early in the day. I'll address these now that I'm in front of a machine. :) > Did they keep writing more Clojure? Yes but it has never been the primary language at Netflix. > How much more did they rewrite from Java to Clojure? Very little, if any was rewritten from Java. > If so, how much of their code is now in Cloj…

>Clojure code bases tend to be much smaller than Java. What are the reasons for this? FP language vs. OOP? Less boilerplate (again maybe due to FP)? Higher-level abstractions in the language or libraries? I have seen that F# code (another FP language, although I've read F# is more from the ML family via OCaml, vs. Clojure being from the Lisp family) can be significantly shorter than equivalent C# code, for example, a…

I worked at a shop that used Clojure and Java.

One big difference is that Java APIs tend to require the collaboration of various class instances to get something done, things that you would implement as a single function + options object on your own.

Bouncy Castle is a good example. You may need a Hasher, HasherStrategy, ASNEncoder, DERParameters, and ASNSerializerStrategy instances to execute what you would've implement as `(asn-encode thing)` otherwise, maybe even having to subclass some of them to change some behavior you'd expect an option flag for.

Clojure's own Java-helper macros will compact your 1:1 Java interop code as well, so you have fewer lines even when writing Java from Clojure. Also, short-cuts like ad-hoc reification in Clojure will spare you LoC where you might otherwise have created a whole file for a class with custom interface implementation in Java.

Of course, line-to-line code is also just more compact in Clojure, but ecosystem/api difference is one I don't see mentioned as often.

I don't think this is just good vs bad, though. There are certainly upsides to the more rigid everything-in-its-right-place code you tend to have in Java which has been making big strides in improving itself over the past decade.

Re: Clojure at Netflix (2013) [slides]

#204
post #80

This is a bit tangential to the post, but I've seen a couple comments in this thread that amount to "Clojure is hard to read", but I think that's a really unfair comparison, because people are unwittingly comparing apples and oranges. It seems to me the reason people think Clojure is hard to read is that the language is so powerful and expressive, that when you're reading Clojure code, you're typically trying to unde…

I worked in a large company where clojure was used. After the original developers moved on from that project (they always move on), it was a giant struggle to get people who could work in the code base. They ended up rewriting it in something more standard. Whatever benifit clojure provided was outweighed by the high cost of maintaining the code base and the inability to find devs who wanted to work on it. That’s my…

If I were seeking a Clojure job I would probably avoid companies that only use it for a single app while 90% of the active development is in more mainstream languages, especially if at any point during the interview process I get a feeling like I was being hired because all the other devs on the project jumped ship.

Re: Clojure at Netflix (2013) [slides]

#205
post #125

One strategy which i learnt from a tech company executive is that if you want to hire best of best at low cost, develop products in interesting new programming languages as the most curious programmers who are very serious about programming care less about money and more about work in a particular language of their interest. No wonder the company had successful projects in every asoteric languages. Now, I come from e…

...if you want to hire best of best at low cost, develop products in interesting new programming languages as the most curious programmers who are very serious about programming care less about money and more about work in a particular language of their interest.

The problem is that the particular language of their interest probably changes every 6 months.

Re: Clojure at Netflix (2013) [slides]

#206
post #162
post #84

Earlier quoted context omitted.

Having spent the last two years working professionally in Haskell after having spent the previous two years working professionally in Clojure, I'd disagree with this. Haskell is roughly infinitely better than Clojure for meaningful refactoring. Clojure gives you (great) tools for using your own brain to make sure the refactor goes well. Haskell replaces your brain almost entirely in the process and just gives you a p…

Just as with the word "developer" [1], I'm beginning to think that the word "refactor" is a dirty word. Are we even talking about the same activity? How is it people are alternately claiming that Haskell and Clojure (virtual opposites on the language spectrum) are infinitely superior to each other at the same thing? Wikipedia defines it as "the process of restructuring existing computer code without changing its exte…

I'd say it comes down to the more standard fare of dynamic vs static typing where the latter has a general advantage in refactoring.

It's true that Clojure has its own perks that aid in refactoring, like how you may have less code altogether. But in my experience it's still a drop in the bucket compared to the zoomed out view of dynamic vs static typing.

For example, could you give some examples that make Clojure particularly good in this regard? I'd have trouble coming up with many that quantify favorably against compile-time analysis. And it would be unfair to take this reality and say "Clojure sucks at refactoring." I think that people do say that is what clouds these discussions where one then needs to point out that Clojure has advantages over other dynamically typed languages which is definitely true.

Re: Clojure at Netflix (2013) [slides]

#207
post #166

Earlier quoted context omitted.

Is Spec simply a test? Or is it something more? I keep hearing about it, but it seems to be more than just a test case. Let's take a simple example. I have an object with the method named "get". But I call "fetch" in my code. When will I see this error? During compile time or run time?

Think of it this way: spec gives you an easy way to ensure that your data conforms to arbitrary predicate functions. Suppose you have an XML format, that has books and authors[1]. The elements look something like this: , . Obviously, you want to make sure that book's author-id attribute will always refer to an author that actually exists. This is something a (Java-style) static type system can't do: it doesn't know a…

>Suppose you have an XML format, that has books and authors[1]. The elements look something like this: , . Obviously, you want to make sure that book's author-id attribute will always refer to an author that actually exists. This is something a (Java-style) static type system can't do: it doesn't know at compile-time what the contents of a variable will be. But spec can do this because it is a runtime check[2]: you'd just write a function that ensures all author-ids refer to extant authors and register it with spec, telling it that this must be true for valid s.

So is it similar to or different from assertions?

Re: Clojure at Netflix (2013) [slides]

#208
post #84

Earlier quoted context omitted.

Having spent the last two years working professionally in Haskell after having spent the previous two years working professionally in Clojure, I'd disagree with this. Haskell is roughly infinitely better than Clojure for meaningful refactoring. Clojure gives you (great) tools for using your own brain to make sure the refactor goes well. Haskell replaces your brain almost entirely in the process and just gives you a p…

To be fair, I wasn't comparing the two. I've never used Haskell (for more than just learning/tutorials). I would suggest that if you have runtime bugs popping up in Clojure programs then that would suggest the inputs to functions (since they should be primarily pure) are not being validated which can easily be accomplished. I would imagine this needs to be done in Haskell as well since just verifying types does not i…

> To be fair, I wasn't comparing the two.

Well, the statement you were disagreeing with from the post you responded to was:

> You can't refactor Clojure without fear like in Haskell.

You go on to suggest you can achieve a similar experience in Clojure by "depending on how you write your code". This simply hasn't been my experience. Just "writing your code the right way" solves almost every problem that arises in programming, but just isn't always feasible in practice (on a team of developers with mixed skill levels, operating under deadlines, etc).

Re. validation, as Matt Noonan mentioned, in Haskell the goal is often to build/leverage correct-by-construction data types which obviate the need for any validation.

A simple example of this would be the `NonEmpty` (list) type.

If you have a function that pulls a list out of some key in a clojure map and it's intended that it always be a non-empty list you still need to check if actually is or not before using it because you have no control over what the caller passes to you. If the caller never sends an empty list you're fine, but if they do and you don't check for it, you've got a bug. Even if you do check for it, there's often nothing sensible you can do at that point since the local function shouldn't know anything about it's calling context, so you have to raise an error or return a nil or something.

On the flip side, in Haskell instead of using a map you would be likely to create a specific data type, and in that data type you would declare the non-empty field to be of the `NonEmpty` type. The first immediate benefit you get is that you no longer have to do any of these checks for the list being empty (or nil, or something else instead of a list) and instead just write your algorithm over the list. Among other things this results in cleaner, simpler, and less code in your immediate function.

But there's another benefit which is now anyone that calls your function has to have constructed a `NonEmpty` list before they call your function. The impact of this essentially naturally propagates the need to construct that `NonEmpty` list to the right place in the code. Maybe it really is just the caller to your your function that needs to take a regular (possibly empty) list that it has and package it up into a `NonEmpty` to call your function... and in that case you still get the benefits of code that shorter, simpler and more clearly communicates it's intention, but where this really shines is when that requirement to pass a non-empty list makes you realize something about the nature of your problem, and you let that `NonEmpty` propagate all the way out towards the boundaries of your application.

Then you end up in a situation where a) all of the code that touches that field anywhere is simpler, clearer, etc. but more importantly b) if someone does send you malformed data with an empty list (say via JSON over a web API or similar) then the code that deserializes the JSON into the `NonEmpty` will fail and you will get an error that says something like "Couldn't decode a YourCustomType from {whatever it was trying to decode}" at the very moment that the bad data tried to enter your system -- instead of just reading the JSON into a map because it was well formed and then letting the record with the empty list in it bounce around until it hits a function that assumes it's non empty at which point you may have little to no information about the provenance of the data or other details that would make easier to solve the problem.

Re: Clojure at Netflix (2013) [slides]

#209
post #70

Earlier quoted context omitted.

With all due respect, I think your argument is a little silly. Typos are the one mistake that programmers never stop making no matter how experienced they are, in my experience. For the same reason, you can pick up a book by an experienced, established writer and find typos in it as well. It's useful for your tools to help you catch basic mistakes like that instead of just saying you should work with programmers who…

I don't think we are talking about that kind of typos. But there is a often a big dip in quality of people, who routinely struggle to write a half decent working program without making these kind of mistakes. Sure every once in a while its possible for any one to drop in a spelling mistake, but those are rare situations in code. Mostly because code is often not a dead piece of documentation, which is read and auto co…

I've deployed tens of thousands of lines of Clojure for a company over several years, and I still find that I occasionally destructure a map with the wrong keyword and I end up with nil accidentally. It's frustrating as much now as it has ever been. Because the app just crashes instead of a simple compiler check that most languages would offer to capture simple stuff like this.

Re: Clojure at Netflix (2013) [slides]

#210

Earlier quoted context omitted.

Did they keep writing more Clojure? Since Netflix likes to constantly write about their tech (videos, blog), my guess would be no, but interested in an official answer too. They are now heavy users of nodejs though. I personally ran out of reasons to prefer another dynamically-typed language over Javascript on the server (not a hard rule of course).

To clarify, the reason I prefer nodejs (for web apps) over clojure these days is the vast amount of libs and documentation for almost all your needs. Clojure has the java ecosystem, but it was very tedious (more cognitive load) investing lots of time doing interop instead of just focusing on the problem at hand. Interop is both a feature and a curse. Of course, as a language I find Clojure superior in almost every wa…

Honestly the vast amount of libs is the reason I don't like nodejs. It's a lot of shoddy duplicates.
Post reply on HN