"How is this better than TensorFlow" is the new "How is this better than Hadoop".
I wonder how their distributed training setup compares to mxent ps-lite (in terms of performance and licensing).
Microsoft Cognitive Toolkit 2.0
31–40 of 53 posts
Re: Microsoft Cognitive Toolkit 2.0
#32They use http://halide-lang.org/ for some convolutions. Very interesting.
Re: Microsoft Cognitive Toolkit 2.0
#33Also, "Reasons to Switch from TensorFlow to CNTK": https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t...
I did not know the below situation with tensor flow.
MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's
"...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public version, or pay to run the TensorFlow job on Google’s cloud. ..."
Re: Microsoft Cognitive Toolkit 2.0
#34The important feature is Keras compatability (although it doesn't seem to be in the official repo yet: https://github.com/fchollet/keras/pull/6800 ) In terms of "how is CNTK better than TensorFlow," CNTK trains 5-10x faster at minimum on pre-2.0 LSTM benchmarks, which is big since LSTMs are used for a lot nowadays. ( https://arxiv.org/abs/1608.07249 )
This is because they use the CuDNN LSTM kernel in CNTK and in TensorFlow they chose to not use the CuDNN LSTM kernel which might be a bit unfair because they could have used it. The same for Torch. See here for more details: https://news.ycombinator.com/item?id=14473234
While it is true that CNTK can use CuDNN LSTM, if the recurrence does not fall into the 4 recurrences that CuDNN supports, CNTK is still much faster. The simplest way to verify this is to take a Keras script that uses whatever recurrent network you want and run it (on a GPU) with Tensorflow backend and with CNTK backend. Some anecdotal evidence suggest an easy 3x speedup.
Disclaimer: I work at Microsoft.
Re: Microsoft Cognitive Toolkit 2.0
#35Also, "Reasons to Switch from TensorFlow to CNTK": https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t...
I did not know the below situation with tensor flow. MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's "...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public ver…
I think that part is misleading. Vijay Vasudevan (a member of the TensorFlow team) has repeatedly put down the notion that the internal TensorFlow code is significantly different than what we see:
https://www.reddit.com/r/MachineLearning/comments/696dzy/d_i...
Obviously, this is taking the word of someone who is incentivized to get as many people using TF as possible, but I haven't been given a reason to not believe him.
I believe that the public TensorFlow repo is very close to what they use internally. That said, I'm sure there is a huge amount of internal tooling (for easily spinning up clusters internally, profiling, probably an automatic device placer) that we don't get to see. But that has more to do with the fact that its designed with Google's specific infrastructure in mind than it has to do with "hoarding the good stuff".
Re: Microsoft Cognitive Toolkit 2.0
#36Also, "Reasons to Switch from TensorFlow to CNTK": https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t...
I did not know the below situation with tensor flow. MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's "...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public ver…
This is baloney. In fact, it's offensive baloney.
There is one TensorFlow. The differences between using TF internally and externally have primarily to do with which RPC bindings it uses (the external one uses gRPC, which is open-source, and the internal one uses the internal RPC framework, which is tied in with all of the internal cluster stuff and authentication and whatnot), and things like filesystems that only exist in Google. The other difference is that there are linkages to use TPUs, instead of just GPUs, which is hardware that doesn't exist outside of Google. The final differences are just in how the BUILD files link against library files -- the external version downloads protobuf for you, the internal version assumes it's there to use. yadda yadda yadda.
You can see all of this in the code. It leaks out in places, such as:
https://github.com/tensorflow/tensorflow/blob/d0d975f8c3330b...
Yes, it's that _super secret_ use of a different integral_types.h header. (/sarcasm). If you look through for things like PLATFORM_GOOGLE in the defines, you'll see a lot of the things that differ, and they're incredibly boring. The core of TensorFlow performance-related stuff is Eigen (or, thanks to Intel's recent contributions, Intel MKL) for executing Tensor ops on CPU, or cuDNN for executing Tensor ops on GPU. Just like every other freakin' framework out there. There's a reason that all of these things tend to reduce to the performance of cuDNN...
See also Pete Warden's article: https://www.oreilly.com/ideas/how-the-tensorflow-team-handle...
("we use almost exactly the same code base inside Google that we make available on GitHub").
(Source: I'm a part-time hanger-on on the Brain team, which develops TensorFlow. I'm also a Carnegie Mellon professor most of the time, and I despise marketing getting in the way of truth.)
Re: Microsoft Cognitive Toolkit 2.0
#37Earlier quoted context omitted.
I did not know the below situation with tensor flow. MS contributions to OSS, at least in this instant, appear a lot more transparent and not-self-centered, compared to Google's "...It was made very clear from the first day of TensorFlow’s announcement, that Google created two TensorFlow versions: a public version and an internal version. As a TensorFlow user, one either must tolerate the slow speed of the public ver…
This is baloney. In fact, it's offensive baloney. There is one TensorFlow. The differences between using TF internally and externally have primarily to do with which RPC bindings it uses (the external one uses gRPC, which is open-source, and the internal one uses the internal RPC framework, which is tied in with all of the internal cluster stuff and authentication and whatnot), and things like filesystems that only e…
Scalability is part of TensorFlow's claimed advantages. If someone adopts TF on their own cluster, would they get the same scalability story as marketed?
Disclaimer: I work at Microsoft.
Re: Microsoft Cognitive Toolkit 2.0
#38The important feature is Keras compatability (although it doesn't seem to be in the official repo yet: https://github.com/fchollet/keras/pull/6800 ) In terms of "how is CNTK better than TensorFlow," CNTK trains 5-10x faster at minimum on pre-2.0 LSTM benchmarks, which is big since LSTMs are used for a lot nowadays. ( https://arxiv.org/abs/1608.07249 )
This is because they use the CuDNN LSTM kernel in CNTK and in TensorFlow they chose to not use the CuDNN LSTM kernel which might be a bit unfair because they could have used it. The same for Torch. See here for more details: https://news.ycombinator.com/item?id=14473234
Why would that be "a bit unfair"? It sounds like the TensorFlow team just hasn't been able to make as many or as good of optimizations as the CNTK team. If they could have, but chose not to do the same things, what tradeoff is there to be able to say it's "a bit unfair"?
Re: Microsoft Cognitive Toolkit 2.0
#39I think their comparison is a bit unfair. > Speed. CNTK is in general much faster than TensorFlow, and it can be 5-10x faster on recurrent networks. This is because they use the CuDNN LSTM kernel but for the TensorFlow comparison they probably did not use the CuDNN LSTM kernel in TensorFlow. See here for some more details: https://news.ycombinator.com/item?id=14473234 > Accuracy. CNTK can be used to train deep learni…
The CNTK C++ API is much more straightforward to use than TF(especially on Windows) and is much better suited to deploying serialized models into production as a part of existing C++ applications. The provided examples require way more boilerplate than an equivalent CNTK application.
Re: Microsoft Cognitive Toolkit 2.0
#40Earlier quoted context omitted.
This is baloney. In fact, it's offensive baloney. There is one TensorFlow. The differences between using TF internally and externally have primarily to do with which RPC bindings it uses (the external one uses gRPC, which is open-source, and the internal one uses the internal RPC framework, which is tied in with all of the internal cluster stuff and authentication and whatnot), and things like filesystems that only e…
Scalability is part of TensorFlow's claimed advantages. If someone adopts TF on their own cluster, would they get the same scalability story as marketed? Disclaimer: I work at Microsoft.
Yes.
Disclaimer: I work at Google.