Live data from Hacker News

Introducing Keras 2

blog.keras.io

61–70 of 70 posts

Re: Introducing Keras 2

#61
post #57

Earlier quoted context omitted.

I tried LightGBM for a Kaggle. I couldn't get anywhere near XGB. I was using the LambdaRank stuff. Given the boasting the LightGBM team had done I had assumed it would be close to XGB out-of-the-box for a ranking problem (since XGB only does pairwise ranking). It was far enough away that I had to ask if I was misinterpreting the output[1]. That was 6 months ago now, so maybe it has improved. I know they made big clai…

Development was rapid when I was working on a blog post in January using the tool. Things have likely improved if you want to give it another shot.

Yeah, I might, thanks.

Did you manage to replicate their results vs XGB?

I don't think anyone has successfully used it for a high result in a Kaggle yet, which - for all its faults - is a good way to see what the maximum performance of a software package seems to be.

LibFFM is the other thing I should have mentioned previously as being worth trying.

Re: Introducing Keras 2

#62
post #59
post #38

Earlier quoted context omitted.

These ops are just not needed in PyTorch. while is just a Python while loop. Scan is a for loop, map is a list comprehension that applies modules. No need for anything fancy.

Sure - but on pytorch they suffer the kernel launch overhead each time through the loop, whereas on tensorflow and theano they do not. Which really impacts the kinds of algorithms that work well on each platform. Does that seem like a reasonable assessment to you?

To not have the kernel launch overhead you'd need to stop launching GPU kernels but that's now how things work in any framework ;)

Re: Introducing Keras 2

#63
post #59
post #38

Earlier quoted context omitted.

These ops are just not needed in PyTorch. while is just a Python while loop. Scan is a for loop, map is a list comprehension that applies modules. No need for anything fancy.

Sure - but on pytorch they suffer the kernel launch overhead each time through the loop, whereas on tensorflow and theano they do not. Which really impacts the kinds of algorithms that work well on each platform. Does that seem like a reasonable assessment to you?

Currently not many frameworks have actual fusion of kernels (to avoid launching many GPU kernels). If you look underneath a theano.scan or TF.scan, GPU kernels are still being launched individually (but are likely stream-overlapped where appropriate).

With TF's XLA compiler, they are slowly getting towards kernel fusion, which will then reduce launch overheads.

We have similar things in the works for pytorch: to quickly JIT at runtime the dynamic graph that is getting executed. More news on this will come when time-appropriate.

Re: Introducing Keras 2

#64
post #63
post #59

Earlier quoted context omitted.

Sure - but on pytorch they suffer the kernel launch overhead each time through the loop, whereas on tensorflow and theano they do not. Which really impacts the kinds of algorithms that work well on each platform. Does that seem like a reasonable assessment to you?

Currently not many frameworks have actual fusion of kernels (to avoid launching many GPU kernels). If you look underneath a theano.scan or TF.scan, GPU kernels are still being launched individually (but are likely stream-overlapped where appropriate). With TF's XLA compiler, they are slowly getting towards kernel fusion, which will then reduce launch overheads. We have similar things in the works for pytorch: to quic…

I WANT to use pytorch, but no bayesian learning or stochastic nodes like in edward. Any chance there are plans to for a compatibility layer with Edward or roll your own bayesian stuff?

Also, have you looked at Numba to do the jitting? Probably best not to have yet another separately maintained python JIT.

Re: Introducing Keras 2

#65

Copying my rare product endorsement from the previous submission: Keras is so good that it is effectively cheating in machine learning, where even Tensorflow tutorials can be replaced with a single line of code. (which is important for iteration; Keras layers are effectively Lego blocks). A simple read of the Keras examples ( https://github.com/fchollet/keras/tree/master/examples ) and documentation ( https://keras.i…

A couple notes on backwards compatibility I ran into yesterday when upgrading auto_ml to use the latest:

Keras now throws errors when trying to use some of the metrics they've deprecated. If you run into `UnboundLocalError: local variable 'class_name' referenced before assignment`, check that all your metrics are supported.

Keras also now ignores `nb_epoch` in favor of `epochs`. I must have misread the blog post, because I thought it would support `nb_epoch` for a bit and just use that in place of `epochs`.

Re: Introducing Keras 2

#66

Copying my rare product endorsement from the previous submission: Keras is so good that it is effectively cheating in machine learning, where even Tensorflow tutorials can be replaced with a single line of code. (which is important for iteration; Keras layers are effectively Lego blocks). A simple read of the Keras examples ( https://github.com/fchollet/keras/tree/master/examples ) and documentation ( https://keras.i…

> Keras is so good that it is effectively cheating in machine learning, where even Tensorflow tutorials can be replaced with a single line of code.

That's the way it "felt" to me when it was introduced in one of the lessons (then later used in a project) in the first term of the Udacity Self-Driving Car Engineer nanodegree I am enrolled in (my cohort just finished the first term; second term for us starts on the 23rd).

We first played around with a simple NN library built from scratch in Python named "Miniflow"; basically, it was a relatively simple backprop library based around Numpy (to eliminate having to implement the vector math part of things). It gave a good overview of how a neural network is developed and implemented at a lower level, and how it actually worked. I say that with having a similar level of knowledge from taking the CS373 course at Udacity (2012) as well as the original (2011) "ML Class" from Andrew Ng - where in both a simple NN was developed using Python and Octave (respectively).

That gave us students the foundation; Tensorflow and how to use it with Python was then introduced (I also took the time and initiative to get my system prepped to use my 750 Ti GPU with TF). That was a revelation - TF made implementing a NN seemingly dead-simple by comparison. It felt almost like plug-n-play!

Then we learned about Keras: I thought that TF made things simple, but Keras proved that wrong. Your comment about it being "Lego blocks" is spot on. It was really simple to implement the NVidia End-to-End CNN to control a virtual car on a track, once it was given proper training from "camera" views.

All that said, though - without having that lower-level foundation of "Miniflow" - you can't appreciate exactly what it is that Keras gives you, nor can you easily grok what is actually happening under the hood, so to speak. I know that our simplified NN library only scratches the surface of what Tensorflow provides, but it does give a foundation on which to experiment and understand things further, IMHO.

Which is why just jumping into Keras without going back to "the roots" of neural networks and machine learning may be doing a disservice to self-learners on this topic. We are still at the point in the process where having the fundamental understandings can help to inform the implementer of solutions in Keras. Its kinda like knowing how to program in Java without understanding what a stack or a linked-list is, or how they work. While it is certainly possible to do so (and produce properly working code), for certain problems having that understanding may be a necessary requirement.

Even when it isn't, though, it still may be a worthwhile thing to know in the end. That's just my opinion, though.

Re: Introducing Keras 2

#67
post #64
post #63

Earlier quoted context omitted.

Currently not many frameworks have actual fusion of kernels (to avoid launching many GPU kernels). If you look underneath a theano.scan or TF.scan, GPU kernels are still being launched individually (but are likely stream-overlapped where appropriate). With TF's XLA compiler, they are slowly getting towards kernel fusion, which will then reduce launch overheads. We have similar things in the works for pytorch: to quic…

I WANT to use pytorch, but no bayesian learning or stochastic nodes like in edward. Any chance there are plans to for a compatibility layer with Edward or roll your own bayesian stuff? Also, have you looked at Numba to do the jitting? Probably best not to have yet another separately maintained python JIT.

as core-devs, we dont plan to build-in something like Edward. However, folks in the community are brewing something:

https://discuss.pytorch.org/t/bayesian-computation-in-pytorc... https://discuss.pytorch.org/t/distribution-implementations/4...

Re: Introducing Keras 2

#68

Earlier quoted context omitted.

You clearly haven't read the book.

Don't need to when you read papers.

You clearly also haven't read many of the papers it cites.

I would say one weakness of the book is that parts of it are too much like a survey of the papers in a subfield. Another is that it is very heavy on theory and light on practice (e.g., no exercises.)

Re: Introducing Keras 2

#69
post #39

Is it better to learn Keras instead of tflearn? Copying a comment I made in another thread where one response recommended Keras: I currently have a small pet project where I think some simple ML would be cool but I don't know where to start. Basically my use case is that I have a bunch of 64x64 images (16 colors) which I manually label as "good", "neutral" or "bad". I want to input this dataset and train the network…

> Is it better to learn Keras instead of tflearn?

Go with Keras. I believe this recent release was partially motivated by Google deciding to fold Keras into tensorflow. Therefore, I would expect keras to supersede tflearn in the areas where they overlap.

https://github.com/fchollet/keras/issues/5050

Re: Introducing Keras 2

#70

Earlier quoted context omitted.

Don't need to when you read papers.

You clearly also haven't read many of the papers it cites. I would say one weakness of the book is that parts of it are too much like a survey of the papers in a subfield. Another is that it is very heavy on theory and light on practice (e.g., no exercises.)

Oh really?

Pray tell me, oh self-conceited one, what I missed that is both in actual use and in that book ? For things outside this set, you'd not read this book anyway; nor would such things be called "deep learning" (other than may be RBMs).

Post reply on HN