Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

github.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#41

Post by waleedka »

taneq wrote: post #34
waleedka wrote: post #20

At a glance: - Only supports fully connected layers for now. No convnets or RNNs. - Requires a GPU. No option to run on CPU, not even for development. - Setup instructions for Ubuntu only. No Mac or Windows. - Uses JSON to define the network architecture. Which limits what you can build. - Takes in data in NetCDF format only. - Very little documentation. - The name is bad. I'm not going to remember how to spell DSSTN…

Why does JSON limit what you can build? Or do you just mean it only supports certain architectures because there are no options to specify other ones in JSON?

Exactly what you said. The declarative approach is great for the common architectures, but if your requirements are different, and the JSON format doesn't have a way to declare it then you're stuck.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#42

Post by scottlegrand »

shoyer wrote: post #39
scottlegrand wrote: post #33

Lead author of DSSTNE here... 1. DSSTNE was designed two years ago specifically for product recommendations from Amazon's catalog. At that time, there was no TensorFlow, only Theano and Torch. DSSTNE differentiated from these two frameworks by optimizing for sparse data and multi-GPU spanning neural networks. What it's not currently is another framework for running AlexNet/VGG/GoogleNet etc, but about 500 lines of co…

Thanks for sharing your story! Let me comment on file formats as someone familiar with both netCDF and deep learning. I agree that netCDF is a sane binary file format for this application. It's designed for efficient serialization of large arrays of numbers. One downside is that netCDF does not support streaming without writing the data to intermediate files on disk. Keep in mind that netCDF v4 is itself just a thin…

Thanks for that! And boy, I wish I had the resources the TensorFlow team has to build standards like this and also to write their own custom CUDA compiler.

I do want the multi-dimensional indexing for RNN data though. Maybe support HDF5 directly is the path forward.

Thanks again!

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#43

Post by barnacle_bill »

throwaway6497 wrote: post #14

Amazon is turning a new leaf. They stopped publishing to any major conferences after their last significant paper, DynamoDB. My perception of Amazon is that they take everything from open-source but don't actively give back. Amazon and open-source never went hand-in-hand. Making their deep learning frameworks open-source is cool. Kudos to the team which managed to do this. I am sure internally, it must have been a hu…

You really think Amazon has something to contribute? A popular thing to do at Amazon is take a complex open source package, wrap it in a web server and announce your team has launched a revolutionary new PAAS. Or take an someone else's web service and build a new web service on top of it with minimal new features and more restrictions. Then announce it and hope for Jeff visibility.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#44

Post by incepted »

waleedka wrote: post #20

At a glance: - Only supports fully connected layers for now. No convnets or RNNs. - Requires a GPU. No option to run on CPU, not even for development. - Setup instructions for Ubuntu only. No Mac or Windows. - Uses JSON to define the network architecture. Which limits what you can build. - Takes in data in NetCDF format only. - Very little documentation. - The name is bad. I'm not going to remember how to spell DSSTN…

> It seems like a very early proof of concept.

Agreed, it looks like a rushed response to TensorFlow.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#45

Post by jrapdx3 »

scottlegrand wrote: post #35
waleedka wrote: post #20

At a glance: - Only supports fully connected layers for now. No convnets or RNNs. - Requires a GPU. No option to run on CPU, not even for development. - Setup instructions for Ubuntu only. No Mac or Windows. - Uses JSON to define the network architecture. Which limits what you can build. - Takes in data in NetCDF format only. - Very little documentation. - The name is bad. I'm not going to remember how to spell DSSTN…

It's more than that, and it's in use in production at Amazon. 8 TitanX GPUs can contain networks with up to 6 billion weights. As Geoffrey Hinton once said: "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain." And you're right that it's a specialized framework/engine. But IMO making it more general purpose is a matter of c…

> "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain."

An interesting quote.

Replicating functioning of the brain, or some major subsystem of it, is no doubt going to require far more than just billions of parameters. The cortex contains >15 billion neurons, but there are also the neurons contained in all the other brain structures. Furthermore, neurons connect via dense dendritic trees, the human brain having on the order of 100 trillion synapses.

Adding to the complexity, neurons have numerous "communication ports", including numerous pre- and postsynaptic neurotransmitter receptors, and a wide range of receptors for endocrine, immune system and other types of signals. Message propagation typically involves as well the layer of complex intracellular "second-messenger" transformations.

While it's highly probably future NNs will be developed that do even more amazing things than now possible, I think the challenge of equaling what real brains do is to say the least enormously daunting.

Somebody smarter than me could probably figure out the magnitude, how many nodes or weights it takes for a NN to function like the brain, though I imagine it will be a really impressive number.

Edit: typos

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#46

Post by waleedka »

scottlegrand wrote: post #35
waleedka wrote: post #20

At a glance: - Only supports fully connected layers for now. No convnets or RNNs. - Requires a GPU. No option to run on CPU, not even for development. - Setup instructions for Ubuntu only. No Mac or Windows. - Uses JSON to define the network architecture. Which limits what you can build. - Takes in data in NetCDF format only. - Very little documentation. - The name is bad. I'm not going to remember how to spell DSSTN…

It's more than that, and it's in use in production at Amazon. 8 TitanX GPUs can contain networks with up to 6 billion weights. As Geoffrey Hinton once said: "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain." And you're right that it's a specialized framework/engine. But IMO making it more general purpose is a matter of c…

Thanks for the clarification. I'd change "early proof of concept" to "a specialized framework", but the other observations stand, I believe.

It's totally fine that it's a specialized framework, and it doesn't need to become general purpose. I just think the product description should do a better job positioning it and explaining what it's NOT intended for to set expectations correctly.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#47

Post by oh_sigh »

meepmorp wrote: post #24
oh_sigh wrote: post #8

Earlier quoted context omitted.

Meanwhile, DSSTNE is completely unmemorable, so even if you wanted to google it, you're going to end up typing "amazon destiny machine learning" or something

I don't know about you, but I'm much more likely to be googling a project as I'm already working with it as opposed to for general information purpose. In that context, "DSSTNE [problem keywords]," seems more useful to me.

That's true... I was mostly thinking about the case where you aren't using it, but remember hearing about it or want to check it out.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#48

Post by xiphias »

scottlegrand wrote: post #33

Lead author of DSSTNE here... 1. DSSTNE was designed two years ago specifically for product recommendations from Amazon's catalog. At that time, there was no TensorFlow, only Theano and Torch. DSSTNE differentiated from these two frameworks by optimizing for sparse data and multi-GPU spanning neural networks. What it's not currently is another framework for running AlexNet/VGG/GoogleNet etc, but about 500 lines of co…

Where do you wok now? It's interesting to hear what offer you couldn't refuse after being in so many places

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#49

Post by fauigerzigerk »

jrapdx3 wrote: post #45
scottlegrand wrote: post #35

Earlier quoted context omitted.

It's more than that, and it's in use in production at Amazon. 8 TitanX GPUs can contain networks with up to 6 billion weights. As Geoffrey Hinton once said: "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain." And you're right that it's a specialized framework/engine. But IMO making it more general purpose is a matter of c…

> "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain." An interesting quote. Replicating functioning of the brain, or some major subsystem of it, is no doubt going to require far more than just billions of parameters. The cortex contains >15 billion neurons, but there are also the neurons contained in all the other brain s…

>Replicating functioning of the brain, or some major subsystem of it, is no doubt going to require far more than just billions of parameters.

Maybe, but we shouldn't forget that computers do not suddenly lose their capability to function as exact, deterministic, programmable machines just because they happen to run an ANN.

What I mean is that there may be shortcuts to reduce the number of required nodes dramatically.

If you take the state of an ANN after it was trained to perform some specific task, you can ask the question whether there is a simpler function, i.e. one with much fewer parameters, that approximates the learned function.

Sort of like a human with the Occam's razor gene. I think the fact that the number of neurons does not correlate perfectly with intelligence in animals is an indication that there is room for optimization.

Re: Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

#50

Post by scottlegrand »

fauigerzigerk wrote: post #49
jrapdx3 wrote: post #45

Earlier quoted context omitted.

> "My belief is that we’re not going to get human-level abilities until we have systems that have the same number of parameters in them as the brain." An interesting quote. Replicating functioning of the brain, or some major subsystem of it, is no doubt going to require far more than just billions of parameters. The cortex contains >15 billion neurons, but there are also the neurons contained in all the other brain s…

>Replicating functioning of the brain, or some major subsystem of it, is no doubt going to require far more than just billions of parameters. Maybe, but we shouldn't forget that computers do not suddenly lose their capability to function as exact, deterministic, programmable machines just because they happen to run an ANN. What I mean is that there may be shortcuts to reduce the number of required nodes dramatically.…

Absolutely 100% agree, but at the same time, I think we will ultimately need to build and evaluate models that can span the memory of more than one processor. I don't think a single GTX Titan X, GTX 1080 or even a server is enough here.

Additionally, data parallelization and ASGD broadly disallow these larger models (yes I know about send/receive nodes in TensorFlow, but they're not general or automatic enough for researchers IMO) while ASGD makes horribly inefficient use of the very limited bandwidth between processors. All IMO of course. There are hacks and tricks here, but I think those should be late stage optimizations, not requirements to achieve scaling.

Finally, I'm a stickler for deterministic computation as someone who spent a decade writing graphics drivers before joining the CUDA team in 2006, but that's pretty much a "hear me now, believe me later" opinion of mine after tracking down too many bizarro race conditions late into the night in that former life :-). Of course, one person's race condition can sometimes be an ANN's regularizer, but I digress.

I also agree we'll do some amazing things with far fewer neurons and weights than an actual human brain, but I'll bet you good money we end up needing more than 12GB to do it. AlphaGo alone was 200+ GPUs, right?

Post reply on HN