Show HN: Eclipse Deeplearning4j
projects.eclipse.org
Show HN: Eclipse Deeplearning4j
1–10 of 41 posts
Re: Show HN: Eclipse Deeplearning4j
#2[deleted]
Re: Show HN: Eclipse Deeplearning4j
#3The interesting thing about this process is timing. You shouldn't really contribute a code base to a foundation until it's mature enough.
I think any group of people developing an open-source project wants it to develop quickly and with healthy governance. Joining any foundation, whether it's Apache or Eclipse or Linux, sends a signal that they're mature, thinking about governance and want to make sure the product develops in a way that agrees with the community.
But sometimes governance can get in the way of speed. What we found talking to Eclipse was that we could get the governance and keep the speed. Which means we'll be able to keep pushing DL4J forward without confusion.
Fwiw, here's the DL4J site: https://deeplearning4j.org/
Here are the repositories: https://github.com/deeplearning4j
And the Gitter community: https://gitter.im/deeplearning4j/deeplearning4j
Re: Show HN: Eclipse Deeplearning4j
#4The interesting thing about this process is timing. You shouldn't really contribute a code base to a foundation until it's mature enough. I think any group of people developing an open-source project wants it to develop quickly and with healthy governance. Joining any foundation, whether it's Apache or Eclipse or Linux, sends a signal that they're mature, thinking about governance and want to make sure the product de…
DL4J strikes me as reasonably mature anyway, although you probably know more about that than I do. :-)
Does this mean anything specific for Skymind as the commercial vendor behind DL4J?
Re: Show HN: Eclipse Deeplearning4j
#5The amount of work we've done this year with Deeplearning4j on performance has been much higher than previous years. We brought DL4J up to par with community standards while maintaining the advantages of Java. I think what a lot of people don't realize is that a ton of effort has been made toward ETL and integration tooling.
It's very difficult to train on multiple GPUs while maintaining performance of ETL. ETL is a scary hidden bottleneck.
I'm very interested to see how Eclipse can continue to push development. I think the people who will especially benefit from this are devops/production teams operationalizing data science.
Re: Show HN: Eclipse Deeplearning4j
#6The interesting thing about this process is timing. You shouldn't really contribute a code base to a foundation until it's mature enough. I think any group of people developing an open-source project wants it to develop quickly and with healthy governance. Joining any foundation, whether it's Apache or Eclipse or Linux, sends a signal that they're mature, thinking about governance and want to make sure the product de…
DL4J strikes me as reasonably mature anyway, although you probably know more about that than I do. :-) Does this mean anything specific for Skymind as the commercial vendor behind DL4J?
That's nice of you. :-) It's mature now, which is why this was the moment to move it into Eclipse.
It means that DL4J & suite is now vendor neutral. On the Skymind side, we will continue to develop all those open-source projects, so you can expect a lot more cool stuff to come: interpretable models, better ETL, Keras as our Python API, vertical-specific apps for EDA, Robotics...
Re: Show HN: Eclipse Deeplearning4j
#7Earlier quoted context omitted.
DL4J strikes me as reasonably mature anyway, although you probably know more about that than I do. :-) Does this mean anything specific for Skymind as the commercial vendor behind DL4J?
That's nice of you. :-) It's mature now, which is why this was the moment to move it into Eclipse. It means that DL4J & suite is now vendor neutral. On the Skymind side, we will continue to develop all those open-source projects, so you can expect a lot more cool stuff to come: interpretable models, better ETL, Keras as our Python API, vertical-specific apps for EDA, Robotics...
Very cool. I'm a DL4J fan. I actually did a talk at Tri-JUG a couple of weeks ago, on real-time machine learning and BPM, which featured DL4J as part of the tech stack. I'm also working on a SaaS offering around AI/ML and plan to include support for DL4J at some point. Hopefully at some point I can get to a place where I can make some useful contributions to the project.
Re: Show HN: Eclipse Deeplearning4j
#8This is super, super cool!
Love the work you guys have been putting out. How do you generally see the community mindshare in space especially with https://techcrunch.com/2017/02/13/yahoo-supercharges-tensorf... (you guys are also mentioned there)
You guys run this at scale, but Yahoo's TensorFlowOnSpark does look very enticing with its "Easily migrate all existing TensorFlow programs with <10 lines of code change" punchline
Re: Show HN: Eclipse Deeplearning4j
#9This is super, super cool! Love the work you guys have been putting out. How do you generally see the community mindshare in space especially with https://techcrunch.com/2017/02/13/yahoo-supercharges-tensorf... (you guys are also mentioned there) You guys run this at scale, but Yahoo's TensorFlowOnSpark does look very enticing with its "Easily migrate all existing TensorFlow programs with <10 lines of code change" pu…
Thanks!
I'm not sure that Yahoo's TF on Spark is fully baked, but maybe its users can tell me more about that. I know it's not commercially supported. That's more a business model distinction, but it has technical consequences. Skymind built the DL4J stack from the ground up, so it's easy for us to maintain and extend, and we give enterprise a way to derisk adoption with a support contract. There is no equivalent offering for TensorFlow outside of GCE, let alone TF on Spark. In the end, it's more useful to think of DL4J as a complement to the Python ecosystem, rather than competing with it. We import TF models through Keras, and soon we'll import them directly to help people productionize the models they train with TF. That's important for large orgs committed to a JVM compute environment.
Re: Show HN: Eclipse Deeplearning4j
#10This is super, super cool! Love the work you guys have been putting out. How do you generally see the community mindshare in space especially with https://techcrunch.com/2017/02/13/yahoo-supercharges-tensorf... (you guys are also mentioned there) You guys run this at scale, but Yahoo's TensorFlowOnSpark does look very enticing with its "Easily migrate all existing TensorFlow programs with <10 lines of code change" pu…
Doesn't matter. We don't need to be the #1 framework. We're going to be focused on model import and integrating with the big data ecosystem.
None of these frameworks you're mentioning integrate properly, are really supported by an actual community, and don't get meaningful updates.
The biggest problem you run in to pretty quickly is maintenance. TF and spark both update quickly.
Every time someone has attempted to do a "TF on spark" they don't add ETL, proper JVM integrations (proper control of cuda and memory management from the JVM), there are also usually strange interactions with JVM/python run times.
Also, please don't ignore the rest of dl4j. We have a whole suite of tools in there. It's not just a matrix lib with autodiff like TF and co is.
TF in its own way is adding some of this stuff like TF records and some of their readers, but it's not going to add a lot in the way of things like connectors to kafka and a lot of the big data ecosystem.
They've basically added HDFS..that was about it.
Dl4j is also significantly easier to deploy. It's just a jar file or zip file..you don't need a blob of c++ just to run a model.