11–18 of 18 posts
I love Julia. I used it to write a restricted Boltzmann machine in less than a page of code ... and my code has lots of whitespace ... and without much effort. It helped that I knew matlab.
I love Julia. I used it to write a restricted Boltzmann machine in less than a page of code ... and my code has lots of whitespace ... and without much effort. It helped that I knew matlab.
I'm curious about your implementation. Can you show us the code?
"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and extensibility: If we use a very high level language that already has a "neural_network_train" function, we can write very concise code but we lose the ability to change the training algorithm. If we use a very low level language that only provides primitive ar…
Whereas a good neural network library like Torch lets you work at a much lower level of abstraction. You can put together individual layers, and it gives you the internal code for doing forward and backward passes, and chaining them together.
"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and extensibility: If we use a very high level language that already has a "neural_network_train" function, we can write very concise code but we lose the ability to change the training algorithm. If we use a very low level language that only provides primitive ar…
As a practical example, brain.js is a very limited neural network library. You can only provide a dataset and train on that dataset. You can't implement any variations like dropout, stochastic gradient descent, momentum, etc. Whereas a good neural network library like Torch lets you work at a much lower level of abstraction. You can put together individual layers, and it gives you the internal code for doing forward…
"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and extensibility: If we use a very high level language that already has a "neural_network_train" function, we can write very concise code but we lose the ability to change the training algorithm. If we use a very low level language that only provides primitive ar…
I think people should be given high level primitives like "layers", allow them to make their own where necessary, but allow the defaults to be: Layer with x (dropout,momentum,..) trained by optimization algo: LBFGS,Hessian Free,.. This allows people to experiment with different configurations without having to dive deep to achieve some basic problems.
Relevant to julia: it's a great language and what I wish production code could look like (while being fast!)
Like rust, it's in a pretty alpha state right now. I'm watching the language heavily though.
Earlier quoted context omitted.
I tried training a 4 layer network in as a young man (because, why not) a day later, no convergence, some calculations revealed that an expectation of many 1000's of days (I have a memory of 1 million, but that could be bs) before the network would converge. So "deep" networks have been around for many decades, and they haven't, because you couldn't train them. Now we have computers that are 10,000* faster (at least)…
This comment nails it. I think it is under-appreciated how much machine learning progress is enabled by increasing computing power. This is not to deny algorithmic improvements, but it's hard to refine algorithms if you can't actually run them.