Deep learning experiments in OCaml

Julia is by no means the only language to have patterns like this either, and in fact it’s not even a language where these patterns are particularly easy to use (I would reserve that for Haskell, but admit there may be other languages I don’t know which also make the cut — not julia though).

Your two ending paragraphs read to me like a super naive restatement of the company line memo for why these types of parametric abstractions are supposed to be good. It’s like a political platform, and just like a political platform it doesn’t keep its promise.

I have worked on projects where we needed to customize bit packing, not for cache performance, but for control over a modified version of sparse matrix types.

And I’m telling you the idea that we’d ever rely on the language’s chosen abstraction and do something like AbstractSparseMatrix{Float64} to pick up a bunch of interface properties “for free” while making the underlying logic specialized for our sparse format is crazy. It’s a naive false promise that grad students believe and it gets quickly beaten out of them in the real world once you realize how the type constraints and inheritance / type class extension constraints this places on you are too limiting and end up requiring just too much boilerplate that can’t quite be autogenerated because the way the abstract interface was chosen just doesn’t quite match your use case.

Finally you realize going down this road was the wrong idea all along, and you just write a super short implementation of MyCustomSparseMatrix or MyCustomCachePropertyMatrix in your case, and you fill in the logic manually that you thought you’d be clever by getting “for free” via plugging into some abstraction hierarchy, and often realize for your use case you don’t need to re-implement hardly any of it, and can do the boring parts pretty easily with converters or helper functions that marshal between whatever “for free” functionality you hoped to get and your simple custom not-parametric-abstraction type.

I’ve been down this road too many times, in many languages. I just leave it for the grad students who like playing with abstraction toys, and instead I just get back to actual work, solving problems economically, which warrants a super strong heuristic of avoiding this type of parametric abstraction pattern as much as possible.

Again, as I’ve been saying, that is a super one-sided way to look at it. Type annotations and the use of appropriate patterns required for most modern “good” static typing are costly things. Organizing things with type class patterns and algebraic data types costs you by making you write more code and more boilerplate, and have trickier things to reason about. Some languages are worse (Scala) about how bad this boilerplate affects you than others (Haskell), but the restrictions it places to facilitate the type-based proofs of safety are real. It’s not free.

Using static typing for these things is only better when (a) the overhead of adding static typing and associated pattern code (and associated maintenance of that extra code and restrictions of how you can write ad hoc code) is not too large and (b) the type-based safety proofs couldn’t have been gotten in some cheaper way.

In a case like a big model training program (a) and (b) just don’t hold. The extra boilerplate and maintenance is very meaningful. Just look at the difference between this blog post’s OCaml code and the equivalent stuff in Keras. The restrictions on ad hoc code also matter. If I can get back a dynamically typed container of settings, like a Python dict for passing into a GPUOptions setup in TensorFlow, and not bother needing to conform to certain types before being allowed to write code that just makes direct assumptions about what attributes I can access, or what dict values will be strings that can serve as args to functions expecting strings, ..., that just saves me lots of time and lets me write way shorter code, relatively speaking, because in this use case it is extremely easy to verify that the only types of data passed in will conform to the assumptions, something that can be checked with an integration test very quickly without requiring any compromise on the handling of arbitrary attributes from config dict values.

Not every case is like this. Some times going to the trouble of setting things up with static typing to prove complicated assumptions are valid within the code is better and ends up reducing code through disciplined use. Static typing can be cost-effective.

This particular use case in the blog post, though, is not one of those cases at all.

And tests are writing themselves for you?

You’d have to write virtually all the same tests in this type of use case whether you are using the static typing approach or not. The tests won’t explicitly check types in the dynamic typing case, but will verify type safety for fixture settings and data indirectly, as a byproduct of all the other testing.

It seems like you are really missing the point. In use cases like a big training program, you have to write integration tests, period. The compiler is not ever a useful substitute for that in this case. Now, since we know you have to use integration tests and so the cost of writing and maintaining those tests is baked in, we can ask: will those tests also cover what a compiler could have helped with, if we’re in the dynamic programming case? Yes.

And so then the extra code we’d have to maintain and extra constraints we’d have to live with if choosing static typing turn out not to buy us anything we can’t already get with the baked-in costs of the integration tests.

Tests don’t write themselves. Why would you ever ask that type of question like you’re being cheeky and rhetorically dramatic? It reveals that you’re still stuck imagining that you’d need to write extra type-specific tests in the dynamic typing case, which misses the point of the discussion.