Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

Post reply on HN
Print view View original

281–290 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#281

Post by CyberSun »

saosebastiao wrote: post #75
codetrotter wrote: post #6

Would be interested to hear their experiences with Clojure at Netflix for the 5 years that have followed since then. Did they keep writing more Clojure? If so, how much of their code is now in Clojure compared to Java? How much more did they rewrite from Java to Clojure? Do they use Clojure rather than Java for new code? What other languages do they use? Python? Erlang? Rust? How much, in terms of functionality, is w…

I noticed a lot of clojure codebases at amazon in 2013 (I was looking because I was learning clojure at the time). It took a look about three years later and most of them were either ported to java or scala, or dead. There were a few still running, but it looked like they were not actively developed or maintained. In my experience at amazon that likely meant they were zombie services (running but nobody used them and…

Our team has been using it pretty consistently for the last three years. There's a few other teams I know of that uses it, but I wouldn't say it's common.

We rewrote more than half our systems from Java to it. We've kept the Java code that was already in good shape, moved the rest to Clojure.

Most of the new systems we've built in the last three years were done in Clojure. Though it's up to the engineer in charge of the project, eventually most everyone on the team has embraced Clojure, and willfully chooses it as their JVM language of choice when starting a new project.

Maintenance wise it has been way better then Java. Even with our team being around 10 devs, which is pretty big. We've had no issues extending our systems, adding features or fixing bugs. We have more stable systems actually, and our tickets count has drastically gone down. Though it's a little unfair, in that some of those systems are rewrites from Java to Clojure, thus our second attempt, and so we might have learned some things from that also.

Most everyone eventually seems to come around to loving it. Like once they are familiar with it enough. At least with regards to Java they like it better, and some who've had Scala experience also seem to prefer it.

All in all, I don't think anyone on the team has any regrets in choosing Clojure, and are actually quite pleased. We're planning to continue to use it, and hopefully have the whole code base be in Clojure.

The weird thing is, you always feel like you wish you had types to help a little with the data flow, and knowing what part to update if you ever change the data model. As well as miss a bit of the auto refactoring tools java provided. But, when we retrospect, it's never really because it's causing us issues like the code base being unmanageable, or refactoring taking us longer to do. It is much more of a feeling, than an actual practical issue. Akin to how driving without your seat-bell on gives you that constant unease of not having it there just in case. That said, we love not having types everywhere adding noise to the code, and the ease of interactivity it offers from the REPL, and being able to focus more on our data model in terms of data (as opposed to types), and focus on our code's behavior (as opposed to types). So its a trade off, that for the last three years, hasn't caused us any issues.

Re: Clojure at Netflix (2013) [slides]

#282

Post by CyberSun »

jatins wrote: post #79
saosebastiao wrote: post #75

Earlier quoted context omitted.

I noticed a lot of clojure codebases at amazon in 2013 (I was looking because I was learning clojure at the time). It took a look about three years later and most of them were either ported to java or scala, or dead. There were a few still running, but it looked like they were not actively developed or maintained. In my experience at amazon that likely meant they were zombie services (running but nobody used them and…

> running but nobody used them and nobody thought to take them down Experienced the same, and these services just kept hoping through the teams. And why is it so hard in Amazon to know if someone is using a service? At one point our manager asked us to shut down a service to see if someone was using it.

Don't know how long it has been that you've left, but this is no longer an issue. There's a discovery mechanism now that keeps track of all service to service interactions.

Also, that's a strange suggestion, couldn't you have just looked at request count metrics?

Re: Clojure at Netflix (2013) [slides]

#283

Post by CyberSun »

Scarbutt wrote: post #199
Scarbutt wrote: post #46

Earlier quoted context omitted.

Did they keep writing more Clojure? Since Netflix likes to constantly write about their tech (videos, blog), my guess would be no, but interested in an official answer too. They are now heavy users of nodejs though. I personally ran out of reasons to prefer another dynamically-typed language over Javascript on the server (not a hard rule of course).

To clarify, the reason I prefer nodejs (for web apps) over clojure these days is the vast amount of libs and documentation for almost all your needs. Clojure has the java ecosystem, but it was very tedious (more cognitive load) investing lots of time doing interop instead of just focusing on the problem at hand. Interop is both a feature and a curse. Of course, as a language I find Clojure superior in almost every wa…

> Clojure has the java ecosystem, but it was very tedious (more cognitive load) investing lots of time doing interop instead of just focusing on the problem at hand.

I suspect you have no JVM/Java experience? Because I do, and I don't in NodeJS, and I find the reverse to be true in my case.

Re: Clojure at Netflix (2013) [slides]

#284

Post by CyberSun »

m12k wrote: post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

Have been using Clojure for three years on a dev team of around 10 people. We haven't had any issue with maintenance and extensibility of the code base over time. I'd say it is as easy to maintain and extend as it was in the beginning.

That said, we have a micro-service architecture. Thus the overall system grows horizontally, in that more and more components are built and developed. So individual components don't really grow that large. This would be different say to building a giant monolith like a AAA game, or a massive application like Photoshop. So I can't speak to how such a code base would scale.

Also, this might sound strange, but I don't think we've ever had to refactor the code base in those three years. The paradigms of Clojure (data driven/functional/immutable/meta), combined with our architecture, it just doesn't really need you to refactor things to add features.

Most adding of features tend to be... Write functions for it. Plug the functions inside the outer orchestration chains of function composition which dictates code flow. Add some more keys to our data-structures. Write tests. Release. Where as in Java, we used to have to shuffle all the Class/Interface arrangements around all the time to make room for new features.

Re: Clojure at Netflix (2013) [slides]

#285

Post by CyberSun »

tanin wrote: post #166
KingMob wrote: post #61

Earlier quoted context omitted.

I use Clojure in the large, and the lack of static typing isn't something I miss. FWIW, spec allows you make similar guarantees if you use it (technically more, since you can express more than HM type systems). I think the static/dynamic divide speaks to deep divisions in programmer personalities, but if you're open to suggestion, I would put it like this: As Haskell/OCaML are to static langs like Java, Clojure is to…

Is Spec simply a test? Or is it something more? I keep hearing about it, but it seems to be more than just a test case. Let's take a simple example. I have an object with the method named "get". But I call "fetch" in my code. When will I see this error? During compile time or run time?

You'd see it at run-time, but most likely at test-time. Which is not a "time" we often think about.

Spec is designed so that you setup automated generative tests on your specced functions. These perform brute force search of the input space.

So, they won't give you FOR ALL guarantees, but will still catch quite a lot. For functions with small input domains, it would actually prove FOR ALL.

The trade off is that, you can test for much more. You can test for properties of the values, not just type. Like say making sure that the output is always smaller than the input. Or that the input never is a blank or empty string.

A downside, it doesn't work well for unpure functions. Since the generative brute force doesn't have a way to brute force the side effect, or assert properties about it. For those, you'd need to write your own tests.

All in all, don't expect it to be at all like a static type checker. It's a very different beast, which you'll want to use very differently, and which offers a very different value proposition. For example, you could want to use Spec even if you had a static type system. Just like people still write tests. Spec is a new kind of tool that can be leveraged to mitigate software defects.

Re: Clojure at Netflix (2013) [slides]

#286

Post by didibus »

bribri wrote: post #276
m12k wrote: post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

This is the number one opposition to Clojure I hear. I was strongly in the static typing camp and rejected Clojure at first, even compared to Java and C++. I tried Haskell and found it amazing, but still too complex. After knowing more about functional programming I returned to Clojure and it finally clicked. I feel writing extremely simple code quickly and reinvesting that time into writing tests was a better workfl…

That's my experience as well.

Static types are overrated. Okay, so you caught the obvious errors a few minutes earlier, but while you were wrestling with types, I ran 30 quick tests validating the behavior, and made them unit tests.

I've only ever had trivial type error bugs, all the nasty ones are behavioral or related to race conditions, or at the boundaries where types don't exist.

Re: Clojure at Netflix (2013) [slides]

#287

Post by didibus »

saosebastiao wrote: post #67
m12k wrote: post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

I've used all three, and at least in my opinion, Scala is the most production ready by miles. Clojure is great if you like dynamic typing...by far the best dynamically typed functional language out there. Beware of claims that clojurescript and clojure are the same language. While both are dynamically typed, clojure's type discipline is strongly enforced (type errors will throw exceptions) while clojurescript will fa…

ClojureScript is strongly typed. I'm curious to know the specific case they ran into? I suspect it might have been interop with JavaScript, because you shouldn't have weak types in ClojureScript.

Re: Clojure at Netflix (2013) [slides]

#288

Post by didibus »

ken wrote: post #162
lgas wrote: post #84

Earlier quoted context omitted.

Having spent the last two years working professionally in Haskell after having spent the previous two years working professionally in Clojure, I'd disagree with this. Haskell is roughly infinitely better than Clojure for meaningful refactoring. Clojure gives you (great) tools for using your own brain to make sure the refactor goes well. Haskell replaces your brain almost entirely in the process and just gives you a p…

Just as with the word "developer" [1], I'm beginning to think that the word "refactor" is a dirty word. Are we even talking about the same activity? How is it people are alternately claiming that Haskell and Clojure (virtual opposites on the language spectrum) are infinitely superior to each other at the same thing? Wikipedia defines it as "the process of restructuring existing computer code without changing its exte…

> The style of refactoring that one does in Haskell (disclaimer: it's been many years since I've written any) is not really possible in Clojure, but it's not really necessary, either.

You're spot on. It just isn't really necessary to refactor Clojure code. In my many years using it professionally, I didn't have to refactor it once. The code just stays clean, it doesn't need cleaning up. So I find the whole refactoring argument moot. Why does your language lend itself to code that needs to be refactored? Sounds like a disadvantage to me.

Re: Clojure at Netflix (2013) [slides]

#289

Post by unlogic »

fiddlerwoaroof wrote: post #180
fiddlerwoaroof wrote: post #178

Earlier quoted context omitted.

Yeah, cider has an amazing form by form debugger for Clojure. And the way cider uses overlays to surface this is quite nice too. My general experience of cider is that for a certain set of tasks it is much better than slime. But, the problem is, most of my day-to-day coding tasks are hampered by the language: e.g. if I have a web server running and I want to change a request handler, you can’t just recompile the hand…

(Although, if you anticipate this, you can call the ref (e.g. `(#’foo arg)`) rather than the function `(foo arg)`. But this means that you have to plan the dynamically modifiable parts out ahead of time.

That's not true. You only have to prepend #' if you pass the function by value (as in your web server handler example).

If you call the function by name yourself, like (foo arg), and you recompile foo, any code that called foo will see the new version.

Re: Clojure at Netflix (2013) [slides]

#290

Post by auganov »

truculent wrote: post #249
auganov wrote: post #150

This talk goes into detail how Clojure plays out in larger teams, particularly how they worked to maintain REPLability for a huge monolith: https://www.youtube.com/watch?v=BThkk5zv0DE That said, I think a lot of the preoccupation with rewrites is just premature optimization. Just a while ago I noticed some open source Clojure code of a startup acquired by FB got rewritten in Haskell. Was very glad to see founders don…

FWIW I think this is the startup code referenced: https://github.com/wit-ai/duckling_old

Yep, that's the one
Post reply on HN