Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tour of our 250k line Clojure codebase

tech.redplanetlabs.com

Post reply on HN
Print view View original

191–200 of 237 posts

Re: Tour of our 250k line Clojure codebase

#191

Post by chromanoid »

xpe wrote: post #134
chromanoid wrote: post #77

> One of the coolest parts of our codebase is the new general purpose language at its foundation. Though the semantics of the language are substantially different than Clojure, it’s defined entirely within Clojure using macros to express the differing behavior. It compiles directly to bytecode using the ASM library. The rest of our system is built using both this language and vanilla Clojure, interoperating seamlessl…

> Actually this sounds quite horrible. You understand the use case well enough to criticize it?

I can only guess regarding the use case. I still can make a statement that I personally do not think that the coolest part of a codebase should be a new general purpose language at its foundation. To me this reeks of not invented here syndrome, inner-platform effect and KISS "violations". IMO when you provide new tooling you want your developers be able to stand on the shoulders of giants and let them choose the fights they want to fight. Idiomatic programming is very helpful in this regard. When programming new languages with macros is idiomatic in Clojure so be it, but I doubt that.

Re: Tour of our 250k line Clojure codebase

#192

Post by cutler »

dustingetz wrote: post #51
davidrupp wrote: post #40

Earlier quoted context omitted.

> in 2007, immutability on the JVM was a competitive value prop, but in 2021+ it is nothing special Can you elaborate on this? What do you think has changed in that time to make it "nothing special"?

immutability as a library is available in basically all mainstream languages now and mainstream frameworks leverage it (react, any UI framework, spark, any data framework or database); JS vms are competitive with the JVM and the JVM might even be losing ground in the cloud; typescript is a monster and is letting people explore haskell concepts in industry applications; scala is way better in 2021 than it was in 2011;…

As far as the JVM goes only Clojure and Scala have real immutability built-in, ie. persistent data structures. Kotlin's default immutability is more like read-only.

Re: Tour of our 250k line Clojure codebase

#193

Post by cutler »

forgetfulness wrote: post #7
cliftonk wrote: post #3

Earlier quoted context omitted.

I've found I typically reach for clojure when i need to do something on the jvm and want a better java than java.

I think that's also why Clojure use peaked right before Java 8 was released; once Java became a better Java, and libraries started to be designed around more ergonomic APIs than wiring up objects that needed miles of stateful configuration code, the pressure that drove you out of Java and into Clojure began to diminish.

Hardly. Despite being Java-compatible Clojure's philosophy is diametrically opposed to that of Java/OOP so anyone who has experienced the benefits of Clojure would hardly return to Java simply because it managed to fake FP with Single Abstract Methods.

Re: Tour of our 250k line Clojure codebase

#194

Post by cutler »

raspasov wrote: post #130
misiti3780 wrote: post #10

If I want to learn Clojure, where is the best place to start? I have a lot of experience with Python/Javascript now, and spent many years in C/C++/Objective C and Java. Also have some Go.

I personally started with this talk by Rich Hickey (person who made Clojure) https://www.youtube.com/watch?v=ScEPu1cs4l0 One of the best "why Clojure" talks that I've seen, especially for somebody coming from an OOP perspective, like myself.

Seconded. Reading books will show you how to write Clojure code but exposing yourself to Rich Hickey's presentations could change your approach to programming in general.

Re: Tour of our 250k line Clojure codebase

#195

Post by mnming »

Really a great article! It's extremely rare to find people who truly understand Clojure/Lisp willing to share to this detail.

Having said that, the article also exposes some flaws of Clojure that I also found.

1. Clojure overly obsesses with expression and non-procedural style coding. While in reality many large scale Clojure repo deploy their own macro to bring procedural style back (like letlocal in the article).

2. The builtin abstraction tools are almost always too simple to be useful.

But those are not big deals IMO.

At the same time, I also wonder if this article mention any feature of Clojure that is truly unique to Clojure compared to other Lisp/Scheme languages. I wonder if the article will still make sense if we simply substitute all "Clojure" to "Racket" (obviously I know ecosystem is not comparable).

Re: Tour of our 250k line Clojure codebase

#196

Post by j-pb »

stevenhuang wrote: post #190
j-pb wrote: post #189

Earlier quoted context omitted.

> Each part of the system can't know what it needs, Yeah it can. It needs what it needed before. If you have a system that is shipping stuff to customers, it doesn't have to care about the new childhoodPetName Field that you've added for a different part of the system. This is especially relevant if you handle any kind of description of the real world, e.g. in robotics or medicine, and have multiple distributed syste…

If field is childhoodPetName and other case labels are e.g., mothersMaidenName, and the common operation is to perform input sanitization, then this new field would have introduced a security hole if it's just ignored. It's a matter of correctness enforced at language level, and why languages like Rust's match statement enforces all checks at compile time by default.

How would it have introduced a security hole?

By definition the field is ignored by your program, therefore even if there is any malicious text in there, e.g. raw html, the system in question can't do anything harmful with it as it is ignored.

Sanitation occurs on a type level, so if the field contains data of the type DangerousRawHTML, and there is a component in your system that just blindly renders any field besides those of type SanitisedHTML, then that is the location of your vulnerability. A type system like Rusts would also catch this.

Don't conflate "correct rust program semantics" with "correct real-world model semantics".

Re: Tour of our 250k line Clojure codebase

#197

Post by barrkel »

TeMPOraL wrote: post #72
habibur wrote: post #65

Earlier quoted context omitted.

> Lots of classes and interfaces but they are all small and with a single purpose. That's the side effect. You ultimately end up with more code and not less even though it saves you from type checks. There are trade offs for both.

One thing I wish for is some kind of "type tags". Being able to express concepts like List[Widget], List[Widget, Nonempty], List[Widget, Nonempty, Sorted], Vector[User, Sorted], etc. - or even, more generic, [ , NonEmpty] (where and are parameters, like in C++ templates) - without implementing an explicit new type for each. Logic verification through typing would then involve not just changing "main" types, but also…

This is quite similar to units (e.g. F# units of measure) or taints (Perl, for tracing user input and detecting unsafe usage).

A large part of the point of these systems is that you can write code which generalizes across the different subtypes (different units, tainted vs untainted values) yet it still passes through invariants by default (values calculated from tainted input are themselves tainted, a unit-quantified value multiplied by a scalar retains its unit, and so on).

I've done similar things with syntax trees in compilers. With multiple passes, you have a representation for the output of the parser, then you might do type annotation, rewrites due to coercion etc., symbol binding and overload resolution, constant expression evaluation, and so on in separate passes. If the input and output trees for each pass have different types, it's easier to keep track of what's going on, of what invariants hold for any given tree node, or indeed which node types are permissible to exist in the input or output of a pass (e.g. you don't want unbound overload calls after overload resolution).

One problem is that tags probably aren't enough when you're dealing with data structures rather than simple zero-dimensional values. Some methods shouldn't be called, or some fields shouldn't be accessed, if a value has a type with the wrong tag. But if you have a wholly different type, then you can end up reimplementing a lot of the same logic, once per type, simply to get the types to flow through.

One solution I've used a couple of times for the tree problem is tree grammars. That is, a data structure which encodes a description of a valid tree and can encode invariants like nodes of type T need to have attributes of type Y with values satisfying predicate P, and between N and M children of type U, V or W, and so on. The grammar is defined in terms of an abstract supertype, and the concrete subtypes have the grammar specific to their phase baked in. This is a hybrid between static and dynamic typing, a compromise necessary for languages like Java without much expressiveness in the type domain.

Re: Tour of our 250k line Clojure codebase

#198

Post by [deleted] »

jwhitlark wrote: post #164
stank345 wrote: post #141

Earlier quoted context omitted.

That's an interesting perspective (and now that you mention it I can think of someone at work who loves types and also tends to break stuff). I myself love static types because it allows me to avoid certain classes of errors and I try my hardest not to introduce bugs. Static typing also greatly informs my workflow. I tend to practice type driven development to the extent possible so when I go back to dynamic language…

Just as bad are those that check in code that that works, but no longer makes logical sense when you read the code. The following is a simple case of what I'm talking about: var flag_is_unset = flag_is_set I generally land more on the dynamic side of things, but there are certainly problem domains where I love types. The more closed and "mathy" the domain, the better I think types fit. I just wish it was less an all-…

[deleted]

Re: Tour of our 250k line Clojure codebase

#199

Post by chromanoid »

xpe wrote: post #132
dd444fgdfg wrote: post #126

Earlier quoted context omitted.

I concur. I think it's nuts when companies do this. There's another well know SaaS that basically invented their own language but I can't recall who. It's the ultimate ego food for the lead engineer imho. what a company needs is good leadership at every level. not unique tools, and not "10x engineers".

> I think it's nuts when companies do this. This can go either way. Yes, it can be abused. DSLs can be very useful and powerful. In any case, I think it is better to not to rush to judgment for this particular case.

I would claim in most cases it is abused. Beside that I criticize the sentiment not the execution.

Re: Tour of our 250k line Clojure codebase

#200

Post by justsomeuser »

chromanoid wrote: post #77

> One of the coolest parts of our codebase is the new general purpose language at its foundation. Though the semantics of the language are substantially different than Clojure, it’s defined entirely within Clojure using macros to express the differing behavior. It compiles directly to bytecode using the ASM library. The rest of our system is built using both this language and vanilla Clojure, interoperating seamlessl…

I thought the same. Clojure is already a niche language, but then you go and niche the niche by implementing another language on top.

Could work out if its essential to offering a 10x service, but I doubt the multiplier is that higher to warrant such an engineering cost of developing and maintaining another language on top of Clojure.

Also, there would be many layers to get to the machine code (Custom lang -> Clojure -> Byte code -> JIT -> Machine code), would this impact performance in a way that makes the program too slow and needs re-writing?

Post reply on HN