Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning experiments in OCaml

blog.janestreet.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Deep learning experiments in OCaml

#71

Post by mlthoughts2018 »

one-more-minute wrote: post #68
mlthoughts2018 wrote: post #36

Earlier quoted context omitted.

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integr…

I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable,…

This is the same for many languages that treat operators with type class patterns. It’s still usually bad design choice. I find a lot of language designers & programmers like it, but they are super disconnected from the realities.

For an example consider breeze and spire in Scala. There’s so much effort to create these bloated numeric type hierarchies that abstract out things like monoids, rings, fields, iterability, sortability, etc.

It’s not good. Just having really boring repetitive implementations for each distinct data structure would be better! No joke! Being able to write type generic functions over sortable matrix subclasses turns out to not be valuable unless you’re also writing a highly abstracted library, which is never, certainly not when you’re using it for experiments.

Nobody needs to be able to make a DenseMatrix[Quaternion] and get it to automatically pick up implementations of fancy indexing. No. You can just write your own helper methods, and this is better, more convenient, applies less pressure for DenseMatrix to have some indecipherably complicated abstract implementation so it can be more free to just specialize on linear algebra functionality that works for DenseMatrix[Double] which is what is needed 99.999999999% of the time.

Re: Deep learning experiments in OCaml

#72

Post by elihu »

mlthoughts2018 wrote: post #4

I had a very unpleasant interview regarding deep learning with Jane Street. I spoke to a member of their HR team to try to get significant assurances that the interview would actually be focused on deep learning and not puzzles or brain teasers, and that the job would really focus on deep learning for their actual business, and not just be a proxy for being generally smart and then work on whatever existing inhouse m…

> I really think this way of thinking about static typing is a very bad thing. This is not at all an actual benefit, because in any sane situation, you will use unit and integration tests that execute extremely quickly on small test data to exercise your end to end model training code.

Unit and integration tests don't write themselves, and they will always be incomplete. You can't test for everything, and what you get from them will depend on how much effort you put in.

Static typing prevents you from running code that tells the computer to do nonsensical things, and usually you'll get an error that tells you exactly what you did wrong. I see those as benefits. In languages like Ocaml or Haskell that have type inference, type annotations can even be omitted most of the time. In the effort-versus-confidence trade-off, I see static typing as low effort with a good payoff. Others might think static typing is too much work and would rather rely on tests. Both approaches are complementary; neither renders the other unnecessary or redundant.

> Turning it around to act like static typing is de facto always a benefit is a very one-sided way to look at it.

Sure, it's a trade-off. Opinions will always vary as to what an ideal productive development environment looks like and what trade-offs are worthwhile, but I think your dismissal of static typing as a tool for gaining some degree of confidence that some program will probably work correctly is also one-sided.

Re: Deep learning experiments in OCaml

#73

Post by ced »

nikofeyn wrote: post #27
flavio81 wrote: post #21

Earlier quoted context omitted.

> It's a bit of a shame that they don't have more widespread use in scientific computing In truth, they have. Lisp was the first functional language (or the first language that allowed that paradigm), and has been used a lot in scientific computing, for example doing symbolic calculus and manipulation.

that doesn't matter when very few scientists have even heard of an ml (standard ml, ocaml, f#) or a lisp/scheme (common lisp, racket), much less have an inkling to use them. their use does of course exist but by no measure is it widespread.

That's part of what makes Julia promising, it's a numerically-focused Lisp, without the parentheses.

Re: Deep learning experiments in OCaml

#74

Post by seanmcdirmid »

shoyer wrote: post #48
yunfeng_lin wrote: post #33

So much bashing on static typing on deep learning:) Does any one from Google can explain the benefit since you guys are working on swift in tensorflow https://medium.com/tensorflow/introducing-swift-for-tensorfl...

Static typing for catching errors is only a small part of the vision for Swift on TensorFlow. The real advantage of static typing is that it enables the compiler to reason to about your code, e.g., to automatically rewrite it for a hardware accelerator with guaranteed correct semantics: https://github.com/tensorflow/swift/blob/master/docs/DesignO... This is obviously possible in Python as well (e.g., see Numba) but i…

Static typing enables some optimizations, but not as many as we’d hope given the inexpressiveness of most type systems.

The real advantage of static typing is code completion, which allows us to forget the nuances of our library naming schemes. TypeScript is so awesome in this regard, being neither sound nor used for optimizations, but still being very useful.

Re: Deep learning experiments in OCaml

#75

Post by jnbiche »

Nelkins wrote: post #24
mlthoughts2018 wrote: post #4

I had a very unpleasant interview regarding deep learning with Jane Street. I spoke to a member of their HR team to try to get significant assurances that the interview would actually be focused on deep learning and not puzzles or brain teasers, and that the job would really focus on deep learning for their actual business, and not just be a proxy for being generally smart and then work on whatever existing inhouse m…

Agreed that type-safety preventing training script failure is not the strongest argument for OCaml. In my experience, the far more compelling reason to use expressive type systems is that they allow you to be more specific about your domain models. This helps to not just prevent type errors (common but not that big of a deal) but also logic errors (I think more common, harder to suss out, and harder to catch with the…

Exactly. When we praise statically-typed languages like OCaml, Scala, and Haskell, I think people miss that we're not just talking about random type annotations. It's exactly what you describe -- we're talking about being able to model domains using the type system, which goes far, far beyond simple type errors.

Algebraic data types are the bedrock for this kind of modeling, and I can't for the life of me understand why more languages don't add them (particularly languages like Java and C#).

Re: Deep learning experiments in OCaml

#76

Post by mlthoughts2018 »

elihu wrote: post #72
mlthoughts2018 wrote: post #4

I had a very unpleasant interview regarding deep learning with Jane Street. I spoke to a member of their HR team to try to get significant assurances that the interview would actually be focused on deep learning and not puzzles or brain teasers, and that the job would really focus on deep learning for their actual business, and not just be a proxy for being generally smart and then work on whatever existing inhouse m…

> I really think this way of thinking about static typing is a very bad thing. This is not at all an actual benefit, because in any sane situation, you will use unit and integration tests that execute extremely quickly on small test data to exercise your end to end model training code. Unit and integration tests don't write themselves, and they will always be incomplete. You can't test for everything, and what you ge…

“You can’t test for everything” seems like a really bad counter argument in this case because you’ll still need to write the integration tests anyway, and if the tests are incomplete (which they always are, it’s life, whether you’re writing statically typed code or not, shrug), you have to improve the tests. You can still have runtime errors and incorrect logic in statically typed code... so?

Now if the process of that testing gets you 99% of the same overall job safety that you’d also get by increasing the code by 10% to add static typing annotations and data structure models (in addition to raising maintenance costs according to that 10% too, and possibly adding bugs or painting yourself into rigid, hard to refactor corners even if they confer some short term bug prevention benefit via the type checking), from tests you already need to write anyway, it’s a no-brainer.

I realize there are good uses of static typing and it can come down to style preference. But truly in this case of “what if my big scientific computing system hits a type-checking-could-prevent-it sort of bug after hours of computing time,” it’s just not a good argument.

This is why people routinely write huge scientific computing systems in Python and nobody ever worries that they hit type checking relevant errors after several hours.

Some things where type checking can really help: ensuring you’ve exhaustively handled every case in an ADT, using the type system to prove state transitions, like with phantom types, using the type system to encode side-effectfulness like Haskell monads.

These things often just aren’t important for something like a large-scale machine learning training program. The types of problems you run into just don’t happen to benefit much from that stuff, while the benefits you can get from writing quick ad hoc functions that can take arguments of unconstrained types and just make unchecked assumptions about their attributes is actually quite big.

Re: Deep learning experiments in OCaml

#77

Post by ChrisRackauckas »

mlthoughts2018 wrote: post #44
dnautics wrote: post #42

Earlier quoted context omitted.

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Ch…

> “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...

How is it premature abstraction if it takes zero extra lines of code to support it and have it optimized? That's kind of the beauty of Julia.

Re: Deep learning experiments in OCaml

#78

Post by ChrisRackauckas »

mlthoughts2018 wrote: post #36
dnautics wrote: post #31

Earlier quoted context omitted.

Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integr…

I'm using the abstract numbers for encodings of uncertainty and probability distributions. This goes far beyond FloatX types and is really helping spawn a whole new area of research. The number type abstractions in Flux are really one of a kind. Kudos to the developers.

Re: Deep learning experiments in OCaml

#79

Post by ernst_klim »

mlthoughts2018 wrote: post #76
elihu wrote: post #72

Earlier quoted context omitted.

> I really think this way of thinking about static typing is a very bad thing. This is not at all an actual benefit, because in any sane situation, you will use unit and integration tests that execute extremely quickly on small test data to exercise your end to end model training code. Unit and integration tests don't write themselves, and they will always be incomplete. You can't test for everything, and what you ge…

“You can’t test for everything” seems like a really bad counter argument in this case because you’ll still need to write the integration tests anyway, and if the tests are incomplete (which they always are, it’s life, whether you’re writing statically typed code or not, shrug), you have to improve the tests. You can still have runtime errors and incorrect logic in statically typed code... so? Now if the process of th…

>“You can’t test for everything” seems like a really bad counter argument in this case because you’ll still need to write the integration tests anyway

Types are nothing more than a proof that some property holds for your code (Curry-Howard correspondence). Tests are nothing but a proof that the property holds for the exact conditions. Types are always better then tests, the only question is how powerful your type system is and how much properties you could express as types. In F* or Idris you don't need tests, in OCaml and Haskell you need tests sometimes, when type system is not powerful enough, in python you have to write tests all the time.

Re: Deep learning experiments in OCaml

#80

Post by dnautics »

mlthoughts2018 wrote: post #71
one-more-minute wrote: post #68

Earlier quoted context omitted.

I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable,…

This is the same for many languages that treat operators with type class patterns. It’s still usually bad design choice. I find a lot of language designers & programmers like it, but they are super disconnected from the realities. For an example consider breeze and spire in Scala. There’s so much effort to create these bloated numeric type hierarchies that abstract out things like monoids, rings, fields, iterability,…

You really should try Julia, before making claims about its complexity.

The numeric type systems are simple, and designed for convenience, not to satisfy mathematical theory. In the case of FloatX, It's basically Any For complex datatypes, like vectors, matrices, dicts, etc, you have templatable datatypes, but that is no more complex than C++, and actually far cleaner in implementation.

For the most part, you do not NEED to make a Matrix{Quaternion}. And that's fine. However, if you do, the standard library will do the right thing, as if you had made a Matrix{Int32} or a Matrix{8BitGaloisField}. And if you choose to use Matrix{Float32}, the type system interacts with the compiler, and in the standard library it picks up the fortran BLAS library so you get faster-than-c performance.

On the other hand, you might be deploying a really large matrix on a supercomputing cluster, and it might be useful to re-index the matrix as a datatype that fits in the L1 cache of your Knights Landing chips. In which case, you have the option of redeploying as an AbstractMatrix{Float64}, implementing index catching functions, and dropping it in to you code (probably about 100 lines of code total, if even) without having to rewrite every single matrix operation everywhere.

Post reply on HN