Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Machine Learning Frameworks

thegradient.pub

Post reply on HN
Print view View original

71–80 of 201 posts

Re: The State of Machine Learning Frameworks

#71

Post by machinelearning »

ehsankia wrote: post #45
machinelearning wrote: post #42

Why engineers like Tensorflow: - More code to check-in (Looks more productive) - More infrastructure, e.g. checkpoints, exporters etc. (Looks like they're doing more work) - Fancy visualizations (Allows them to look impressive while presenting loss plots) - Easier to reuse things others have implemented and still get credit for it (TF model zoo, research repo etc.) Why researchers like pytorch: - Way easier to hack t…

Researcher's codes are historically not very clean and re-usable. It may work fine if you want to hack together something to get data for a paper, but if you want to run a real production service, and don't want to drown in tech debt in a year; and that often means more code, as you imply above. I don't think it's unnecessarily verbose, it's just that it's more structured and scalable.

The discussion was about pytorch vs tensorflow but your argument seems to be about productionizing research code (which is framework agnostic).

Yes there is a lot of code to be written/optimizations to be done to make things production worthy. However I know a lot of tensorflow research projects that handle data batching in such a terrible way that would take weeks to re-write for production.

As for verboseness of pytorch vs tensorflow, I think either could get more verbose under different circumstances. However for simpler tasks, I think tensorflow is more verbose in general (not accounting for the new release which seems to mimic pytorch/keras a little more). For larger production tasks, its a toss-up depending on whether you need to add new components.

Re: The State of Machine Learning Frameworks

#72

Post by machinelearning »

andbberger wrote: post #54
machinelearning wrote: post #42

Why engineers like Tensorflow: - More code to check-in (Looks more productive) - More infrastructure, e.g. checkpoints, exporters etc. (Looks like they're doing more work) - Fancy visualizations (Allows them to look impressive while presenting loss plots) - Easier to reuse things others have implemented and still get credit for it (TF model zoo, research repo etc.) Why researchers like pytorch: - Way easier to hack t…

Speak for yourself mate. Why I use tensorflow: - keras - I used tensorflow yesterday

What's your experience comparing keras to pytorch?

Re: The State of Machine Learning Frameworks

#73

Post by arugulum »

Granted that PyTorch and TensorFlow both heavily use the same CUDA/cuDNN components under the hood (with TF also having a billion other non-deep learning-centric components included), I think one of the primary reasons that PyTorch is getting such heavy adoption is that it is a Python library first and foremost.

There're maybe all of two "surprises" I've encountered in all my time using it, if even (1. Gradients are accumulated in state, 2. nn.Module does funky things with attributes, so use something like nn.ModuleDict if you're going to be dynamically setting modules). Everything else works like a dream, and works almost exactly how you expected.

Model parameters? .parameters() gives you a dict-friendly generator of tensors. Model state? .state_dict() is a dictionary. Loading model state? load_state_dict(state_dict)... just loads a dictionary. Reusing modules across different modules? Just assign them! Determining what parameters to optimize? Just ... give the list of parameters to the optimizer.

You can use all your using Python development and debugging tools, and it feels 100% natural. I can fit it into other Python workflows without making the whole program centered around TensorFlow.

TensorFlow is undoubtedly powerful, and if you have the time/resources to put into a static-ish TensorFlow-centric workflow, it could pay off many times over. But it definitely feels like learning an entirely new language, with an entirely different debugging pattern. And furthermore, a language that is constantly changing patterns and best practices, other than super-standard Keras examples.

To put into context, even running the official TensorFlow models repository has deprecation warnings. Whereas torchvision works like seamlessly and reads like a reference for writing PyTorch model code.

There is just a developer-centric focus to PyTorch that makes it a joy to use.

Re: The State of Machine Learning Frameworks

#74

Post by chillee »

Der_Einzige wrote: post #47

I don't even believe the thesis about one dominating the other in a specific domain. I dont think top mentions in conferences in a good measure of usage.

Why not?

Ok, admittedly, there are a couple reasons. The fact that most papers don't mention the framework they use is a big one. So if users of one framework disproportionately mentioned that framework in their paper, it would be overrepresented.

I did cover this concern though, in the Appendix. Check out the "Biased Sample" section.(https://thegradient.pub/p/cef6dd26-f952-4265-a2bc-f8bfb9eb1e...)

Basically, some conferences have encouraged researchers to submit code. Instead of checking the papers, I checked their code instead. The results are pretty much the same. So I think that mentions in top conferences probably correlates well with uses in code.

Re: The State of Machine Learning Frameworks

#75

Post by _coveredInBees »

microtonal wrote: post #52

We are considering to move to PyTorch, we really dislike how the Tensorflow 1.x -> 2.0 transition is handled. For years a lot of stuff has been added to tf.contrib , some things were only in tf.contrib and now that it's dropped in TF a lot of project (including ours) have to do quite large rewrites. Since the last few 1.x iterations, Tensorflow has been complaining that the older RNN layers are deprecated and that we…

It's been pretty good with PyTorch. The API has been fairly stable and I've adopted code developed from 0.4.0 to 1.0.0+ with barely the need for any tweaks. Granted, it's a younger project so for now things are stable but maybe 3 years from now they may have some giant API refresh. But I find their API quite nice for the most part so I don't see them needing to switch everything up periodically.

Re: The State of Machine Learning Frameworks

#76

Post by _coveredInBees »

UncleOxidant wrote: post #62
_coveredInBees wrote: post #19

We use Pytorch extensively in our startup. We tackle a lot of new research problems as consultants/partners to help develop products or devise new algorithms/models to solve tasks for our customers. We have never regretted our choice to pick Pytorch. I found the article pretty spot on when comparing Tensorflow and Pytorch. The things that have appealed to me about Pytorch are: 1. Extremely easy to debug and work with…

Last year I was tasked with looking into a NAS (Neural Architectures Search) paper and analyzing the algorithm. The paper came with a TensorFLow implementation. Trying to read that TF code was quite difficult. I searched around and found a PyTorch implementation - much easier to read and understand, and it ran about 50% faster as well (the latter was a bit surprising). I tend to think that TensorFlow lends itself to…

Yes, installing Pytorch with Cuda support has been unbelievably simple. I've never had trouble with simply pip installing it in my venv.

Re: The State of Machine Learning Frameworks

#79

Post by abakus »

mlthoughts2018 wrote: post #67

Keras still is the very best in terms of expressive models and end to end workflows. It leans heavily on the design idea that you should deliberately design for end to end use cases and all intermediate abstractions should be building blocks that serve precisely that purpose. This is discussed in [0] which IMO is something that deserves to be more widely talked about in software engineering. Lots of other disciplines…

In terms of model expressiveness, I made a functional NN building API for PyTorch (just like keras'), which offers the optimal balance of flexibility and expressiveness: https://github.com/blue-season/pywarm

Re: The State of Machine Learning Frameworks

#80

Post by tsbinz »

pillefitz wrote: post #26
amrrs wrote: post #9

I think tensorflow dominates industry purely because of its capability of exporting the model into a coreml Android model or easy of moving it to production in a GCP environment or in whatever form. Pytorch might have to build a good production pipeline around it to catch up in this game. With fastai module that's built on Pytorch learning and developing Deep Learning solutions have become a lot easier. So there's a…

You can do the same with PyTorch, right? Just export the model as .onnx and import it with whatever inference engine you like.

This sounds like in theory and makes you curse in practice. Trained TF model -> deployment using TF Lite is the most robust pipeline currently as far as I know.

There are lots of improvements going into pytorch for mobile at the moment, but for the moment I'll wait and see how it turns out - I didn't have much fun with caffe2 when "train in pytorch and deploy with caffe2" was the storyline FB pushed (e.g. problems with binary size and slow depthwise convolutions) so not too eager to migrate back at the moment.

Post reply on HN