Live data from Hacker News

Clojure at Netflix (2013) [slides]

speakerdeck.com

251–260 of 307 posts

Re: Clojure at Netflix (2013) [slides]

#251

Earlier quoted context omitted.

Did they keep writing more Clojure? Since Netflix likes to constantly write about their tech (videos, blog), my guess would be no, but interested in an official answer too. They are now heavy users of nodejs though. I personally ran out of reasons to prefer another dynamically-typed language over Javascript on the server (not a hard rule of course).

To clarify, the reason I prefer nodejs (for web apps) over clojure these days is the vast amount of libs and documentation for almost all your needs. Clojure has the java ecosystem, but it was very tedious (more cognitive load) investing lots of time doing interop instead of just focusing on the problem at hand. Interop is both a feature and a curse. Of course, as a language I find Clojure superior in almost every wa…

You can use ClojureScript on top of Node. My team started using Node for some services, and I published Macchiato based on that https://macchiato-framework.github.io/

Re: Clojure at Netflix (2013) [slides]

#252

I always see loads of Static vs Dynamic and Functional vs OO opinions breaking out in the comments. While the discussion is often interesting and insightful, it would be nice to see some empirical analysis of the trade-offs here. Stuff like development time, error rates, maintainability (no idea how you'd measure this one...). Anyone know if such studies exist? Feels like they should be out there given the amount of…

There is some research available on the topic. There's a good summary of it here https://danluu.com/empirical-pl/

Re: Clojure at Netflix (2013) [slides]

#253
post #55

I'd like to learn a modern production-ready functional language (I have some academic experience with SML), since they seem like a good way to grow as a programmer. Main contenders so far are Haskell, Scala and Clojure (Reason might go on the list soon too) - but the fact that Clojure is dynamically typed is a bit of a turn off for me. My experience with dynamic vs static typing is that as a system grows in size and…

Worked with static typing for about a decade primarily with Java in the enterprise. However, I've also used Haskell and Scala which have advanced type systems. I moved to work with Clojure about 8 years ago, and I don't miss types. If I did, I would've gone back to a typed language a long time ago.

My experience is that dynamic typing is problematic in imperative/OO languages. One problem is that the data is mutable, and you pass things around by reference. Even if you knew the shape of the data originally, there's no way to tell whether it's been changed elsewhere via side effects. Keeping track of that quickly gets out of hand.

What I find to be of highest importance is the ability to reason about parts of the application in isolation, and types don't provide much help in that regard. When you have shared mutable state, it becomes impossible to track it in your head as application size grows. Knowing the types of the data does not reduce the complexity of understanding how different parts of the application affect its overall state.

My experience is that immutability plays a far bigger role than types in addressing this problem. Immutability as the default makes it natural to structure applications using independent components. This indirectly helps with the problem of tracking types in large applications as well. You don't need to track types across your entire application, and you're able to do local reasoning within the scope of each component. Meanwhile, you make bigger components by composing smaller ones together, and you only need to know the types at the level of composition which is the public API for the components.

REPL driven development [1] also plays a big role in the workflow. Any code I write, I evaluate in the REPL straight from the editor. The REPL has the full application state, so I have access to things like database connections, queues, etc. I can even connect to the REPL in production. So, say I'm writing a function to get some data from the database, I'll write the code, and run it to see exactly the shape of the data that I have. Then I might write a function to transform it, and so on. At each step I know exactly what my data is and what my code is doing.

Where I typically care about having a formalism is at component boundaries. Spec [2] provides a much better way to do that than types. The main reason being that it focuses on ensuring semantic correctness. For example, consider a sort function. The types can tell me that I passed in a collection of a particular type and I got a collection of the same type back. However, what I really want to know is that the collection contains the same elements, and that they're in order. This is difficult to express using most type systems out there, while trivial to do using Spec.

[1] https://vvvvalvalval.github.io/posts/what-makes-a-good-repl.... [2] http://danboykis.com/?p=2293

Re: Clojure at Netflix (2013) [slides]

#254
post #208

Earlier quoted context omitted.

> To be fair, I wasn't comparing the two. Well, the statement you were disagreeing with from the post you responded to was: > You can't refactor Clojure without fear like in Haskell. You go on to suggest you can achieve a similar experience in Clojure by "depending on how you write your code". This simply hasn't been my experience. Just "writing your code the right way" solves almost every problem that arises in prog…

Exactly, and I was responding to the fact that I never fear refactoring my Clojure code. Your examples of the type safety can all be mimicked with spec in Clojure. Granted spec is opt-in (but I'm guessing so is some of the more detailed type safety attributes you are talking about like NonEmpty). Anyhow, to each their own and one persons experience isn't likely to be the same as the others so I'd encourage everyone t…

The opt-in vs opt-out nature is definitely key. As an example, if you write a small function to operate on a non-empty list and use it on a non-empty list in exactly one place, everything is fine in either language. But in Haskell if someone else tries to use the function passing in an empty list, it won't compile, they will encounter the problem right away. In Clojure if someone sees the function and says "oh good, someone's already written this function, I'll just use it" and doesn't realize that it expects a non-empty list, then you now have another situation where something is going to explode when data that contains an empty list arrives. And the error is again going to be far (in the code and in time) from the source of the problem. The difference between the experience of "I have good tools that I can use effectively to arrange to avoid bugs" and "I have tools that simply will not allow entire (large) classes of bugs at all" is what I'm really trying to get at. This is just an area where Clojure shines compared to most dynamic languages but Haskell shines compared to nearly all languages.

Re: Clojure at Netflix (2013) [slides]

#255
post #123
post #99

Earlier quoted context omitted.

Try to grok REPL-driven development before dynamic typing. You won't like the latter until you love the former.

Haskell does REPL-driven development fairly well. It's not the best REPL, but it works, especially if you take a moment to learn a few of the commands to drive it. Dynamic types are not necessary for REPL-driven development.

Haskell REPL is a completely different animal from the one in Clojure, and it's a stretch to say that you can do REPL driven development in Haskell.

What people mean by REPL driven development in context of Clojure is that the REPL is running within the context of the application you're developing. The editor is connected to the REPL, and you can modify any part of the application at runtime. You can even connect the editor to a REPL on a remote machine such as a production server and inspect its state from your editor.

Re: Clojure at Netflix (2013) [slides]

#256
post #168

Earlier quoted context omitted.

I'd disagree with this. Primarily been developing with Clojure for 5 years now with some pretty large codebases. It does depend on how you write your code but favoring pure functions, pushing immutability to the edges of your programs allows you to refactor without fear. Clojure also has many things to aid in this such as pre/post conditions, clojure.spec (which allows you to build complex type definitions), and of c…

> It does depend on how you write your code but favoring pure functions, pushing immutability to the edges of your programs allows you to refactor without fear. Dynamic typing is punitive if we make mistake (e.g. using the wrong architecture, now we have to refactor), and we'll make mistakes. Writing perfect code with perfect architecture and perfect test has been the argument "for" dynamic typing, but I don't think…

My experience is that static typing tends to lead to higher coupling and monolithic design, while you tend to break things up more aggressively in a dynamic language like Clojure.

Re: Clojure at Netflix (2013) [slides]

#257
post #140

Earlier quoted context omitted.

Yes, that's why becoming happy with REPL-driven development is more important than becoming happy with dynamic typing. I think much of the productivity boost attributed to dynamic typing is in fact caused by heavy use of a REPL, and not dynamic typing.

I'd maybe consider that perhaps becoming happy with REPL-driven development isn't a universal solution. It's not like I didn't understand the concepts, the tools, or the process. I just didn't find that it made me any more proficient at building software.

Of course that's possible, but the evidence suggests that most people become more productive when using the REPL to experiment while developing large projects.

One issue with trying to adopt the habit midway through a project is having a design that makes it easy to load small portions of the project with test data into the REPL. It's not quite the same as having code that's easy to test.

Do you use an interactive debugger? It's the same principle, but earlier in the lifecycle of development.

Re: Clojure at Netflix (2013) [slides]

#258
post #257

Earlier quoted context omitted.

I'd maybe consider that perhaps becoming happy with REPL-driven development isn't a universal solution. It's not like I didn't understand the concepts, the tools, or the process. I just didn't find that it made me any more proficient at building software.

Of course that's possible, but the evidence suggests that most people become more productive when using the REPL to experiment while developing large projects. One issue with trying to adopt the habit midway through a project is having a design that makes it easy to load small portions of the project with test data into the REPL. It's not quite the same as having code that's easy to test. Do you use an interactive de…

I appreciate your earnest approach, but I also feel somewhat patronized. I've been doing this for a while. I used Clojure extensively for years, was productive and found much to like. The REPL wasn't part of it, and I don't miss it. I still use Clojure on occasion.

This response is pretty common: "You probably weren't doing it right. Trust me, this is the way!" While I agree that suspending beliefs/old practices and trying new things is important when using new technology, I'm also pretty confident that I gave Clojure a more than fair shake, and just find that I don't get much out of the technology surrounding the REPL in that environment.

I was more productive in Scala in days than I was after years of using Clojure, and the difference between what I'm saying and your response is that I'm not claiming my experience is universal.

Re: Clojure at Netflix (2013) [slides]

#259

Earlier quoted context omitted.

Explicit runtime checks that must be manually added are not a substitute for static typing! You should follow your own advice and try out a statically typed language with good inference! Every tool has a sweet spot and large code bases and maintenance is well outside the sweet spot of any dynamic language.

Well, there are certainly classes of problems that I'd use static typing for, but in my experience, static typing != software that gets the job done. That doesn't mean that it holds things back, though. Many static typing arguments remind me of the Air Force's old "We'll bomb them so hard we won't have to send in ground troops." I just haven't seen it in practice, and the few studies that have looked at it empiricall…

> I just haven't seen it in practice, and the few studies that have looked at it empirically haven't seen a clear advantage either. If you know of such a study, please point it out!

This is basically right; there are not a ton of studies, and the ones that exist mostly have pretty bad methodologies. Dan Luu summarized a bunch of them, circa about 2014. [1]

Since then, there has been one study in this area that I think has a solid, well-defined, and plausible methodology [2]. Plus a "Threats to Validity" section, sorely missing from many other papers in this area. They work from a corpus of real-world public bugs in Javascript programs and quantify how many are detected by simple type annotations via TypeScript and Flow. They cap the amount of time for trying to resolve a bug with type annotations at 10 minutes. The result is that over a corpus of 400 bugs, they were able to resolve about 60 using either of TypeScript or Flow, suggesting that 15% of Javascript bugs can be eliminated by using either type system.

That's not a huge difference, but it isn't trivial either. The authors quote an engineering manager at Microsoft: "That’s shocking. If you could make a change to the way we do development that would reduce the number of bugs being checked in by 10% or more overnight, that’s a no-brainer. Unless it doubles development time or something, we’d do it."

I suspect with languages that are more amenable to static types the results would be even better, but there is no solid research that I know of to back that up.

[1] https://danluu.com/empirical-pl/

[2] http://ttendency.cs.ucl.ac.uk/projects/type_study/documents/...

Re: Clojure at Netflix (2013) [slides]

#260

Earlier quoted context omitted.

To clarify, the reason I prefer nodejs (for web apps) over clojure these days is the vast amount of libs and documentation for almost all your needs. Clojure has the java ecosystem, but it was very tedious (more cognitive load) investing lots of time doing interop instead of just focusing on the problem at hand. Interop is both a feature and a curse. Of course, as a language I find Clojure superior in almost every wa…

You can use ClojureScript on top of Node. My team started using Node for some services, and I published Macchiato based on that https://macchiato-framework.github.io/

I like the idea behind Macchiato but unlike it's Clojure cousin, Luminus, there doesn't seem to be any documentation on database access. I especially like HugSQL in Luminus. Is it possible to use it with Macchiato?
Post reply on HN