Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning experiments in OCaml

blog.janestreet.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Deep learning experiments in OCaml

#61

Post by yunfeng_lin »

KenoFischer wrote: post #57
shoyer wrote: post #48

Earlier quoted context omitted.

Static typing for catching errors is only a small part of the vision for Swift on TensorFlow. The real advantage of static typing is that it enables the compiler to reason to about your code, e.g., to automatically rewrite it for a hardware accelerator with guaranteed correct semantics: https://github.com/tensorflow/swift/blob/master/docs/DesignO... This is obviously possible in Python as well (e.g., see Numba) but i…

Static typing has very little to do with what the compiler can say about your code. You can have dynamic languages with very strong type systems and semantics as well as static languages with weak semantics. The only difference between static and dynamic languages is whether the compiler enforces completeness of the analysis or not.

I am not sure I agree with you. You do need compile time type to generate efficient hardware accelerated code.

Python has strong type, but that is only available at run time, which is not useful to generate code.

But now python also have optional type. this might be utilized in generating more efficient code though

Re: Deep learning experiments in OCaml

#62

Post by KenoFischer »

yunfeng_lin wrote: post #61
KenoFischer wrote: post #57

Earlier quoted context omitted.

Static typing has very little to do with what the compiler can say about your code. You can have dynamic languages with very strong type systems and semantics as well as static languages with weak semantics. The only difference between static and dynamic languages is whether the compiler enforces completeness of the analysis or not.

I am not sure I agree with you. You do need compile time type to generate efficient hardware accelerated code. Python has strong type, but that is only available at run time, which is not useful to generate code. But now python also have optional type. this might be utilized in generating more efficient code though

Look at e.g. Julia for a dynamic system with a strong type system that allows the compiler to reason about the code without forcing completeness.

Re: Deep learning experiments in OCaml

#63

Post by checkyoursudo »

senorsmile wrote: post #39

Am I the only one who gets confused by references to ML (ML derived typed FP vs Machine Learning)? The threads on this page are the represent a strange junction where I really have to think about what people mean, because they really could mean either!

ML with ML?

I occasionally have to double check, yes.

Re: Deep learning experiments in OCaml

#64

Post by yunfeng_lin »

KenoFischer wrote: post #62
yunfeng_lin wrote: post #61

Earlier quoted context omitted.

I am not sure I agree with you. You do need compile time type to generate efficient hardware accelerated code. Python has strong type, but that is only available at run time, which is not useful to generate code. But now python also have optional type. this might be utilized in generating more efficient code though

Look at e.g. Julia for a dynamic system with a strong type system that allows the compiler to reason about the code without forcing completeness.

Julia has optional typing which is static as well

Re: Deep learning experiments in OCaml

#65

Post by KenoFischer »

yunfeng_lin wrote: post #64
KenoFischer wrote: post #62

Earlier quoted context omitted.

Look at e.g. Julia for a dynamic system with a strong type system that allows the compiler to reason about the code without forcing completeness.

Julia has optional typing which is static as well

As always in these sorts of discussions, it depends how you define your terminology. However, by most definitions of static typing, Julia's type system is not static. The julia type system is very much a property of the runtime language and behaves as such. It is quite strong, true, but still not statically enforced as you would expect from a static language. In particular, (to the extent that you can identify one), you never get any sort of compile-time type errors in julia.

Re: Deep learning experiments in OCaml

#66

Post by elbear »

deepGem wrote: post #15
mlthoughts2018 wrote: post #11

Earlier quoted context omitted.

Static typing is also a nice way to communicate design intentions. But for this to work, the annotations have to be very expressive. I don’t know the first thing about OCaml, but I have worked professionally with Haskell and static typing is a joy when it adds clarity and makes the contracts of functions instantly readable. Contrast this with Scala, which I have also worked with professionally and the difference is s…

Yep agreed. Though you can communicate design intentions in comments, no ? There is another argument that programmers might not follow the comments, so strict enforcement by types helps - I don't believe in such a philosophy though. Most programmers will do the right thing and mistakes are not intentional.

It's not that they don't follow the comments, it's the fact that comments aren't executable, so they rot. People forget to update them. That can't happen with type definitions, because your code stops compiling.

Sure, comments serve their purpose, but that purpose only slightly overlaps with that of static types.

Re: Deep learning experiments in OCaml

#67

Post by 3rdAccount »

remify wrote: post #58
mi_lk wrote: post #16

Earlier quoted context omitted.

You know him personally? Or he has some prior works that we can take a look?

Ahah no, I was just looking at his CV and the guy is jacked.

Hahaha! This is a comment I would normally make and be scoffed at.

I'm always amazed at how smart some people are.

Re: Deep learning experiments in OCaml

#68

Post by one-more-minute »

mlthoughts2018 wrote: post #36
dnautics wrote: post #31

Earlier quoted context omitted.

Flux is great. Because it's Julia, I could write a custom datatype that has fewer bits and test to see if inference and training are possible, and then apply that datatype to ml models without writing custom kernels (except convnets, but I'm going to push code for that.)

This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes. However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill. Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integr…

I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable, but actually more work than just leaving it generic.

We didn't at any point decide "it's worth the extra effort/complexity to make Flux work with custom number types"; it's just inadvertently been that way from day one, and I didn't even know anyone one was making use of it until today.

Re: Deep learning experiments in OCaml

#69

Post by mark_l_watson »

Very nice. I have spent many evenings playing with the Haskell bindings for TensorFlow that don’t have the coverage these OCaml bindings have (e.g., character seq models).

I have thought of learning some OCaml, maybe this will give me the kick in the butt to do it.

Re: Deep learning experiments in OCaml

#70

Post by throwawaymath »

deepGem wrote: post #8
mlthoughts2018 wrote: post #4

I had a very unpleasant interview regarding deep learning with Jane Street. I spoke to a member of their HR team to try to get significant assurances that the interview would actually be focused on deep learning and not puzzles or brain teasers, and that the job would really focus on deep learning for their actual business, and not just be a proxy for being generally smart and then work on whatever existing inhouse m…

The type safety argument is total BS. First of all the training script will fail for the very first time if there is a type error. You'd be a moron to pass an argument of a different type 'a couple of hours' into the training. No sane programmer writes such code. What kind of nonsensical argument is this. What I have found static typing to be really useful for is in remembering what I have coded. It's quite hard to r…

> What I have found static typing to be really useful for is in remembering what I have coded. It's quite hard to remember a dynamic type while you are writing code, given the number of variables you are dealing with. Seeing that type definition next to your variable name is a handy reference.

If you consider this a benefit, then (for example with Python), don't you get the same benefit just by using docstrings? Stated another way, if all you want is a visual cue about what you're passing to a function and getting returned, why bother with all the scaffolding of type safety? You can get that just by using documentation facilities outlined in various languages' style guides. Those language facilities (such as docstrings) tend to be very useful and a good engineering practice in general.

The point of type safety is actually to obviate what you're talking about. Smart developers can and do make the mistakes you're saying only a moron would make, regardless of available visual cues. Offloading that decision making process to a language that complains when you make that mistake instead of being forgiving about it is entirely the point.

So I guess what I'm saying is that I'm struggling to understand why you think type safety is BS. If I read you correctly, it sounds like you'd also say that developers committing memory corruption vulnerabilities are morons, and that the scaffolding of memory management and garbage collection is BS. Why not just have explicit references a developer can read while coding to make sure they're not overflowing a container, right?

Post reply on HN