Live data from Hacker News

Tour of our 250k line Clojure codebase

tech.redplanetlabs.com

171–180 of 237 posts

Re: Tour of our 250k line Clojure codebase

#171
post #84

> One of the coolest parts of our codebase is the new general purpose language at its foundation. Though the semantics of the language are substantially different than Clojure, it’s defined entirely within Clojure using macros to express the differing behavior. It compiles directly to bytecode using the ASM library. The rest of our system is built using both this language and vanilla Clojure, interoperating seamlessl…

Imagine starting a job as a closure dev and being told you have to learn a new language with substantially different semantics

I think it really comes down to how well the new semantics fit the problem domain. If it's a much better fit than the alternative, it's a net win. But just because you can define new semantics, doesn't make it a good idea.

This is Nathan Marz we're talking about though; he's on the short list of people who can both accurately identify the need and create something workable to fill it. If he told me it was necessary for a particular problem domain, I (personally) would think long and hard before I disagreed.

Re: Tour of our 250k line Clojure codebase

#172

Earlier quoted context omitted.

Yes, and this is where statically typed languages shine (in my opinion). I like programming in a style that makes heavy use of the type system to enforce this. For example when writing an api endpoint to create a task I would typically deserialise the json into a CreateTaskRequest. If the object is created without exceptions I can be sure it is valid. CreateTaskRequest implements the ToTask interface. The service lay…

I think it is always important to understand that the data models and the approaches to modelling the world are totally different between Clojure and strongly typed languages. I’m going to ignore C++ and Java style languages because I think they give a Clojure-style model of the world without any of the benefits of a well-suited programming language or a type system that can enforce new invariants. In the ML-family o…

> Fundamentally, I think the differences stem more from philosophies about modelling the world than type systems.

It seems like people mistake the battleground for the countries. If you only saw the US and Japanese battles in early World War II, you'd think they were tiny nations of ships, soldiers and aircraft fighting over a handful of islands. It wouldn't be clear that there were full countries with big populations backing all this, and that they were thousands of miles apart.

So Rich Hickey gives a talk criticizing types, and a Haskeller responds with a blog post, and you think that types are the difference between the camps. But it's not. It's just that this is the spot that's close enough to home to reach, but contentious enough to fight over.

Re: Tour of our 250k line Clojure codebase

#173
post #84

> One of the coolest parts of our codebase is the new general purpose language at its foundation. Though the semantics of the language are substantially different than Clojure, it’s defined entirely within Clojure using macros to express the differing behavior. It compiles directly to bytecode using the ASM library. The rest of our system is built using both this language and vanilla Clojure, interoperating seamlessl…

Imagine starting a job as a closure dev and being told you have to learn a new language with substantially different semantics

I know people who work at RPL, am 1000% confident that anyone who joins knows _exactly_ what they are getting into (and this kind of crazy Nathan Marz rabbit hole is exactly why they signed up).

Re: Tour of our 250k line Clojure codebase

#174

Earlier quoted context omitted.

My concern with all-or-nothing validation of the sort you're suggesting is that sometimes the invalid data is data that isn't your concern and doesn't matter to the task you're trying to accomplish. It doesn't even have to be a mistake on the sender's side; maybe something evolved, maybe you made the mistake. I prefer defining the minimum I need for a specific function, instead of defining exactly what I expect. For…

When encoding types for requests like this, I'm exactly as strict as I need to be: no more, no less. Any assumption about the shape of the data that my code will make is encoded in the parser, but the parser is very generous with everything else. For example, if a request has a field I didn't expect, that's not an error, I simply ignore that field because I obviously didn't need it. Likewise if requests start using a…

That sounds like it might work. I was commenting based on my experience of people deciding there had to be a single representation for an idea/type no matter what it was used for, across multiple projects and repositories. I was not dismissing appropriate validation at a boundary, just trying to pointing out that, at least in my experience, people get way too wound up about enforcing unnecessary consistency.

Re: Tour of our 250k line Clojure codebase

#175
post #159

Earlier quoted context omitted.

you concluded something is impossible based on the fact that you have never seen it before? talking about silly claims :)

Yes, that's how observation and personal opinion works.

well it should not be how personal opinions are formed - maybe you should look up first principle thinking if you have not yet. I do not know if their 100X claim will come true or not, however saying something is impossible merely because it has not been done in the past is clearly wrong

Re: Tour of our 250k line Clojure codebase

#176
post #175

Earlier quoted context omitted.

Yes, that's how observation and personal opinion works.

well it should not be how personal opinions are formed - maybe you should look up first principle thinking if you have not yet. I do not know if their 100X claim will come true or not, however saying something is impossible merely because it has not been done in the past is clearly wrong

> saying something is impossible

Point out where I said "impossible"? I said skeptical, which is an absolutely perfectly position to take. I'm all perfectly happy to be wrong.

Re: Tour of our 250k line Clojure codebase

#177
post #2

This is what I imagine experienced clojure developers can squeeze out of a language like clojure. I would venture that they can train a junior programmer in a couple of weeks, and make them productive very fast. I guess they could make it work in any language but judging by the description clojure is indeed a great fit, due to the macro capabilities, flexibility and solid runtime via JVM.

What I heard from colleagues that work with Clojure is that it is a horrible language where the default way of writing code is an imperative programming style where contexts are passed around and updated. Far from the concepts of functional programming.

Well it's a bit preposterous of you to say that without actually having tried Clojure in good faith.

But, since we all make opinions from others, let me provide you a balance of opinions by giving you mine.

I'm a senior engineer and I currently use Clojure professionally. I find it to be a lovely, fun and productive language, my favorite one to date actually. I have prior professional experience with C++, C#, ActionScript 3, JavaScript, Scala, Kotlin and Java. And of all of those, Clojure is my favourite to work with.

The "context' pattern you may have heard of, I believe I know what it refers too, and it is not an imperative pattern at all, let me explain.

There's often a case where a piece of functionality will be implemented by multiple functions composed together. For example, an API handling a request will delegate to sub-functions which could in turn call down to more sub-functions.

In that scenario, it can happen that the sub-functions need data about the context of the call to the parent function, or they need data generated by the prior sub-functions that were called. In my example, lets say the request object to the API dictates various options and has the request details, and each options is relevant to different sub-functions, maybe one needs the username, where the other needs the cart-id both of which were passed on the request, but maybe the other function also needs the user-permissions which was obtained from the prior sub-function.

Ok, so say:

    someAPI({username: "foo", cart-id: 123}) {
      permissions = get-permissions(username);
      retrieve-cart-items(username, permissions);
}

Now as you have more and more of these forming a coherent whole, in Clojure there is a pattern where people will say, all this initial data and the data generated by the intermediate steps becomes the "context" data for the operation as a whole.

    (defn some-api [context]
      (-> get-permissions
          retrieve-cart-items
          :cart-items))
Where context is at first a map of the request data:

    {:username "foo",
     :cart-id 123}
And after the call to "get-permissions" it is a map of the initial request data and the intermediate data added by get-permissions:

    {:username "foo",
     :cart-id 123,
     :permissions [:can-view-cart]}
And after the call to retrieve-cart-items it now also contains the cart-items:

    {:username "foo",
     :cart-id 123,
     :permissions [:can-view-cart]
     :cart-items [:pen, :paper]}
Thus all implementing functions for the some-api operations are designed so they take an initial context map and return a context map with added context.

And generally in Clojure you'd use the destructuring syntax to indicate what in the context map you depend on:

    (defn get-permissions
      [{:keys [username]:as context}]
      ...)
So you see that get-permissions expect the :username key to be present on the context.

But, this is still fully functional, because none of the functions share a mutable reference to a shared context, they take an immutable context as input and return a new immutable context as output which will just happen to also include all the key/values of the context they received.

Sometimes people also pass in dependencies using this context pattern, which is a form of dependency injection through parametrization, but you combine all dependencies into one parameter object.

Re: Tour of our 250k line Clojure codebase

#178

Earlier quoted context omitted.

Can Kotlin handle multiple "tags" on a type as a set, and not a sequence? I'm not familiar with the language, so I'll use a C++ analogy. If you tried to tag types in C++, you'd end up with something like: TaggedType but such type is strictly not the same as: TaggedType What I mean by "set" instead of a "sequence" is to have the two lines above represent the same type, i.e. the order of tags should not matter.

You can maybe get there in kotlin with a generic type with multiple constraints in a where clause. Let’s say you have Sorted and NonEmpty as interfaces (could be empty marker interfaces so they behave like tags). Then you can write a method fun doSomething(values: T) where T: Sorted, T: NonEmpty {} And that function will take any type that has both Sorted and NonEmpty interfaces.

You don't need Kotlin to do this, even Java can do it:

     void doSomething(T values)

Re: Tour of our 250k line Clojure codebase

#179
post #154

> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem. This is a great insight no matter what language or framework you're operating in. Laziness has its virtues, but invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible - it's much easier to debug issues whe…

> invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible Having worked on numerous large-scale distributed systems, I strongly disagree (and I would think the author would too). There is a tradeoff to strictness, which is coupling. You want to validate and type check as much as necessary—but not more. If you are over-coupled in a distributed system, it can mak…

> Each part of the system should validate what it needs

Easier said than done. Each part of the system can't know what it needs, since you've just added a new thing it might need to contend with.

The point of closed enums is to force these checks, even if all you're doing is adding a new fall through to ignore it, since at least that means you've considered it.

Re: Tour of our 250k line Clojure codebase

#180
post #154

> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem. This is a great insight no matter what language or framework you're operating in. Laziness has its virtues, but invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible - it's much easier to debug issues whe…

> invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible Having worked on numerous large-scale distributed systems, I strongly disagree (and I would think the author would too). There is a tradeoff to strictness, which is coupling. You want to validate and type check as much as necessary—but not more. If you are over-coupled in a distributed system, it can mak…

It's not a matter of distributed systems, but a matter of systems where only parts of it are updated on prod at the same time. You could easily imagine a huge release, where the whole codebase gets pushed to prod at the same time. Then, the whole issue of versioning APIs disappears. However, it would require more discipline (probably unachievable at Google scale), so most companies prefer the slowly-burning garbage fire of versioned APIs and backward/forward compatiblity of messages.
Post reply on HN