Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning experiments in OCaml

blog.janestreet.com

Post reply on HN
Print view View original

41–50 of 99 posts

Re: Deep learning experiments in OCaml

#41

Post by dnautics »

xvilka wrote: post #37
phonebucket wrote: post #17

Earlier quoted context omitted.

I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without go…

Julia is very good, the only problem is the requirement for patched LLVM (patches they provide are not yet merged in the upstream), which can cause the conflicts with other frameworks if there is no separation.

This is fairly easily dealt with using containers.

Re: Deep learning experiments in OCaml

#42

Post by dnautics »

mlthoughts2018 wrote: post #36
dnautics wrote: post #31

Earlier quoted context omitted.

Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integr…

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?

There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype).

This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed.

In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.

Re: Deep learning experiments in OCaml

#43

Post by gaius »

habitue wrote: post #40
gaius wrote: post #10

Type-safety helps you ensure that your training script is not going to fail after a couple hours because of some simple type error. This isn’t a failure mode that ever happens in DL... 2 hours into the job you will only be dealing with floats anyway no matter what language you are using. If you’re going to fail on anything typed it will be in the first 20 seconds probably, basically the instant you start your first e…

This is true, but it's also because Tensorflow is a typed language, but it uses the syntax of python. (At least the default way) in tensorflow, the graph is type checked on startup, and it'll fail if anything is wrong. Contrast this with pytorch, chainer, or tensorflow's dynamic computation graphs and they're much more likely to have a bug that happens later, since their graphs aren't verified up front. Unfortunately…

I admit I don’t have much experience with those, I’m a Keras and CNTK guy but the principles will be the same: marshal your data into a huge matrix of floats/one-hot and hand it off to training where it will spend 99.9% of it’s time.

I am a fan of strong/static typing and was once very active in the OCaml community but that just struck me as a very odd thing for the OP to say... it’s just not something that people doing DL worry about. It could be valuable in the marshalling phase but that all happens before DL begins and (in my experience) in a separate program.

Re: Deep learning experiments in OCaml

#44

Post by mlthoughts2018 »

dnautics wrote: post #42
mlthoughts2018 wrote: post #36

Earlier quoted context omitted.

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integr…

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Ch…

> “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?”

This sounds like premature abstraction to me...

Re: Deep learning experiments in OCaml

#45

Post by yaseer »

senorsmile wrote: post #39

Am I the only one who gets confused by references to ML (ML derived typed FP vs Machine Learning)? The threads on this page are the represent a strange junction where I really have to think about what people mean, because they really could mean either!

This happens to me too, having dabbled with ML for theorem proving.

Thing is, ML is an obscure language for most people. The association with machine learning probably dominates in 95% of people.

Re: Deep learning experiments in OCaml

#46

Post by icc97 »

yaseer wrote: post #45
senorsmile wrote: post #39

Am I the only one who gets confused by references to ML (ML derived typed FP vs Machine Learning)? The threads on this page are the represent a strange junction where I really have to think about what people mean, because they really could mean either!

This happens to me too, having dabbled with ML for theorem proving. Thing is, ML is an obscure language for most people. The association with machine learning probably dominates in 95% of people.

It's becoming less obscure, F# (.net), Elm and Reason (JavaScript) are bringing ML to a wider audience. Plus Jane Street do a great job of promoting the use of OCaml.

Re: Deep learning experiments in OCaml

#48

Post by shoyer »

yunfeng_lin wrote: post #33

So much bashing on static typing on deep learning:) Does any one from Google can explain the benefit since you guys are working on swift in tensorflow https://medium.com/tensorflow/introducing-swift-for-tensorfl...

Static typing for catching errors is only a small part of the vision for Swift on TensorFlow. The real advantage of static typing is that it enables the compiler to reason to about your code, e.g., to automatically rewrite it for a hardware accelerator with guaranteed correct semantics: https://github.com/tensorflow/swift/blob/master/docs/DesignO...

This is obviously possible in Python as well (e.g., see Numba) but is clearly has additional challenges: https://github.com/tensorflow/swift/blob/master/docs/WhySwif...

(I work at Google, but not on the TensorFlow team.)

Re: Deep learning experiments in OCaml

#49

Post by lordnacho »

nikofeyn wrote: post #29
burkaman wrote: post #12

Earlier quoted context omitted.

I applied for a more entry level software position at Jane Street, and while I would have failed the interview regardless, I had the same experience where the HR person had no idea what the interview was like. They assured me I'd be required to write OCaml, so I spent the weekend brushing up, and that I should bring my own laptop prepared with whatever development I wanted to use. In fact it was a couple "whatever la…

i wouldn't sweat it. people design those interviews to make them feel better about themselves rather than as an effective way to gauge someone's aptitude to work there. you'll notice that none of these places that ask these types of questions allow the candidate to ask them technical questions. it's always a one way street. there's times where i have "failed" interviews of this type when i could guarantee i have "sim…

> i wouldn't sweat it. people design those interviews to make them feel better about themselves rather than as an effective way to gauge someone's aptitude to work there.

Same reason I dislike that sort of interview. Of course the thing to do is to throw in a spanner.

"Hmm, so I guess you didn't read about the Modified Banach-Wiles-Kolmogorov algorithm? I thought that was where we were going. Ok let's do it your way."

Throw this bomb on the way out, of course.

Re: Deep learning experiments in OCaml

#50

Post by phonebucket »

mlthoughts2018 wrote: post #44
dnautics wrote: post #42

Earlier quoted context omitted.

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then? There is no complexity added in flux to support arbitrary datatypes; Ch…

> “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?” This sounds like premature abstraction to me...

Premature abstraction is a problem for engineering.

But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.

Post reply on HN