Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning experiments in OCaml

blog.janestreet.com

Post reply on HN
Print view View original

31–40 of 99 posts

Re: Deep learning experiments in OCaml

#31

Post by dnautics »

phonebucket wrote: post #17
mlthoughts2018 wrote: post #14

Earlier quoted context omitted.

I would also suggest looking into Keras and PyTorch too. I think they honestly achieve a greater degree of elegance and a greater degree of mapping the programming constructs into the mental model space of the domain expert, than any FP interface to neural nets that I’ve seen yet.

I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without go…

Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)

Re: Deep learning experiments in OCaml

#32

Post by rkido »

typon wrote: post #28
mlthoughts2018 wrote: post #4

I had a very unpleasant interview regarding deep learning with Jane Street. I spoke to a member of their HR team to try to get significant assurances that the interview would actually be focused on deep learning and not puzzles or brain teasers, and that the job would really focus on deep learning for their actual business, and not just be a proxy for being generally smart and then work on whatever existing inhouse m…

I've always thought that static typing is the least interesting thing about functional languages.

Not all function-oriented languages use static binding. Due to how the BEAM VM works, Elixir/Erlang is very, very function-oriented but also late-bound. Nevertheless, its type system is relatively rich compared to most other late-bound languages.

It's never static binding that makes a language good or interesting, apart from performance benefits. You can get the productivity boon of early-warning type-checking with or without it. What makes a language good, in my opinion, is its ability to provide a type system that closely matches the needs of the domain in which that language is to be used. For example, game development requires a data-oriented approach, which Rust's type system practically forces the developer to adopt.

More complex software engineering problems require more expressive type systems. But unless you're the one writing TensorFlow, machine learning is insignificant from a systems engineering perspective; it's simple enough for non-programmers. Thus, expressive type systems don't seem to offer much benefit here.

Re: Deep learning experiments in OCaml

#34

Post by sincerely »

nikofeyn wrote: post #29
burkaman wrote: post #12

Earlier quoted context omitted.

I applied for a more entry level software position at Jane Street, and while I would have failed the interview regardless, I had the same experience where the HR person had no idea what the interview was like. They assured me I'd be required to write OCaml, so I spent the weekend brushing up, and that I should bring my own laptop prepared with whatever development I wanted to use. In fact it was a couple "whatever la…

i wouldn't sweat it. people design those interviews to make them feel better about themselves rather than as an effective way to gauge someone's aptitude to work there. you'll notice that none of these places that ask these types of questions allow the candidate to ask them technical questions. it's always a one way street. there's times where i have "failed" interviews of this type when i could guarantee i have "sim…

I don't think I've ever had an interview where it would have been okay to quiz the interviewer on technical topics...could you explain a little better?

Re: Deep learning experiments in OCaml

#36

Post by mlthoughts2018 »

dnautics wrote: post #31
phonebucket wrote: post #17

Earlier quoted context omitted.

I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without go…

Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes.

However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill.

Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integration hurdle to make it recognized by the backend, and then requires published research or some similar type of evidence that there are use cases which materially benefit from that new additional fixed data type, to get a PR approved to add it.

The reason is that permitting arbitrary complexity growth in the form of “easy” custom data type support has two big downsides, (a) the mechanism that makes it easy had to consume maintenance and development resources even if it’s a very obscure form of customization, and (b) more importantly, it proliferates and worsens the already insane problems of being able to export / import models from one language/framework to another.

It’s a case study of KISS and YAGNI: this is super premature abstraction especially if it’s for experiments. And the hurdle of making a branch and adding your new dtype in the backend is not (and should not be seen as) a significant engineering hurdle. Rather it’s a very good check on complexity growth.

Re: Deep learning experiments in OCaml

#37

Post by xvilka »

phonebucket wrote: post #17
mlthoughts2018 wrote: post #14

Earlier quoted context omitted.

I would also suggest looking into Keras and PyTorch too. I think they honestly achieve a greater degree of elegance and a greater degree of mapping the programming constructs into the mental model space of the domain expert, than any FP interface to neural nets that I’ve seen yet.

I use PyTorch a lot; it's definitely my preferred framework at the moment. I just wish there was something as thoughtfully done and well-supported in a more functionally oriented language. Flux.jl on Julia is the frontrunner in this regard, IMO. The added benefit is that being written in Julia the whole way down makes it easy for practitioners to delve into the source code and extend it in a performant way without go…

Julia is very good, the only problem is the requirement for patched LLVM (patches they provide are not yet merged in the upstream), which can cause the conflicts with other frameworks if there is no separation.

Re: Deep learning experiments in OCaml

#38

Post by nikofeyn »

sincerely wrote: post #34
nikofeyn wrote: post #29

Earlier quoted context omitted.

i wouldn't sweat it. people design those interviews to make them feel better about themselves rather than as an effective way to gauge someone's aptitude to work there. you'll notice that none of these places that ask these types of questions allow the candidate to ask them technical questions. it's always a one way street. there's times where i have "failed" interviews of this type when i could guarantee i have "sim…

I don't think I've ever had an interview where it would have been okay to quiz the interviewer on technical topics...could you explain a little better?

there's a couple things. why is it not okay? as a potential employee, you want to be sure that they have an inkling as to what they are doing and aren't just asking questions out of the "interviewing at google" book. i am not actually arguing that interviewees ask these questions (see below), but it is something to think about. i usually try to ask some questions that give me a sense of how big of a mess their software is. at one of the places i failed the final interview, i wasn't ever shown any code or product that they were working on but had a sense it was a bit of a mess with a bunch of smart people writing code with no hands at the wheel.

i have the perspective that these types of puzzle questions by the interviewer are pointless. and i was getting at the point that the interviewee asking similar pointed questions would be similarly useless. because it's easy to take that high and mighty stance instead of having a conversation. it creates an artificial environment that doesn't really exist in actual working environments.

and i generally feel that companies are far too arrogant in their hiring process. they very much create a one way dialog as if you should be thanking them for even interviewing you. they act like "we don't need you, you need us". it creates a very bad taste in my mouth, and even if i was to be offered employement by such companies, it is possible i would turn them down unless i am convinced that their work environment is distinctly different from their interviewing process.

Re: Deep learning experiments in OCaml

#39

Post by senorsmile »

Am I the only one who gets confused by references to ML (ML derived typed FP vs Machine Learning)? The threads on this page are the represent a strange junction where I really have to think about what people mean, because they really could mean either!

Re: Deep learning experiments in OCaml

#40

Post by habitue »

gaius wrote: post #10

Type-safety helps you ensure that your training script is not going to fail after a couple hours because of some simple type error. This isn’t a failure mode that ever happens in DL... 2 hours into the job you will only be dealing with floats anyway no matter what language you are using. If you’re going to fail on anything typed it will be in the first 20 seconds probably, basically the instant you start your first e…

This is true, but it's also because Tensorflow is a typed language, but it uses the syntax of python. (At least the default way) in tensorflow, the graph is type checked on startup, and it'll fail if anything is wrong.

Contrast this with pytorch, chainer, or tensorflow's dynamic computation graphs and they're much more likely to have a bug that happens later, since their graphs aren't verified up front.

Unfortunately, typed languages won't help you much there. A big reason people use pytorch is because of its flexibility (i.e. they were bumping up against the constraints of a static graph system and wanted out)

Post reply on HN