Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Singa, a Distributed Deep Learning Platform

singa.incubator.apache.org

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Apache Singa, a Distributed Deep Learning Platform

#11

Post by jkovacs »

vosper wrote: post #2

There are so many Apache projects these days (mostly seem to be in the incubator) - where do they all come from? I had once thought that Apache was a dumping ground for the technology of failed startups, but a lot of these projects seem to find new life in the incubator - so they must be doing something right.

Lots of these (including this one) come out of universities, or university-corporate research partnerships. Personally I'm very glad that universities nowadays seem to be more okay with publishing tangible results (i.e. source code) as opposed to "just" the research papers. I don't think it's always been that way. I can't begin to imagine the amount of work that is rotting on someone's old usb stick, because they weren't allowed to publish their PhD theses's source code.

Re: Apache Singa, a Distributed Deep Learning Platform

#12

Post by gojomo »

modeless wrote: post #7
glxc wrote: post #5

Earlier quoted context omitted.

if your data exceeds memory then a GPU is worthless. distributed makes it scalable

This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed. Furthermore, almost any tec…

At least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limits of GPUs:

http://research.google.com/archive/large_deep_networks_nips2...

Re: Apache Singa, a Distributed Deep Learning Platform

#13

Post by igul222 »

modeless wrote: post #7
glxc wrote: post #5

Earlier quoted context omitted.

if your data exceeds memory then a GPU is worthless. distributed makes it scalable

This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed. Furthermore, almost any tec…

This is also not quite right. Models whose parameters are too big to fit on one GPU can be trained by splitting them across multiple GPUs, as was done here, for example: http://papers.nips.cc/paper/5346-sequence-to-sequence-learni...

Re: Apache Singa, a Distributed Deep Learning Platform

#14

Post by limau »

modeless wrote: post #7
glxc wrote: post #5

Earlier quoted context omitted.

if your data exceeds memory then a GPU is worthless. distributed makes it scalable

This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed. Furthermore, almost any tec…

SINGA provides an abstraction for defining all known models,a neural net structure that is easy for model and data partitioning, and a parallelism model that supports both synchronous, asynchronous and hybrid training frameworks. Processing at each node could the be CPU, GPU, or CPU-GPU based.

Re: Apache Singa, a Distributed Deep Learning Platform

#15

Post by monksy »

AlphaSite wrote: post #3
vosper wrote: post #2

There are so many Apache projects these days (mostly seem to be in the incubator) - where do they all come from? I had once thought that Apache was a dumping ground for the technology of failed startups, but a lot of these projects seem to find new life in the incubator - so they must be doing something right.

The list of big name Apache projects is nuts, look at their project lists and you can easily pull out dozens of big projects and even incubator has a few popular projects.

It's hard to find whats worthwhile and what isn't.

Re: Apache Singa, a Distributed Deep Learning Platform

#16

Post by brianchu »

igul222 wrote: post #13
modeless wrote: post #7

Earlier quoted context omitted.

This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed. Furthermore, almost any tec…

This is also not quite right. Models whose parameters are too big to fit on one GPU can be trained by splitting them across multiple GPUs, as was done here, for example: http://papers.nips.cc/paper/5346-sequence-to-sequence-learni...

According to the paper, the parameters fit on one GPU (or at least that one GPU was able to train the model). It was just too slow, so they trained on 8 GPUs in parallel. But those GPUs were still on the same machine (one node, multiple GPUs).

Re: Apache Singa, a Distributed Deep Learning Platform

#17

Post by mindcrime »

vosper wrote: post #2

There are so many Apache projects these days (mostly seem to be in the incubator) - where do they all come from? I had once thought that Apache was a dumping ground for the technology of failed startups, but a lot of these projects seem to find new life in the incubator - so they must be doing something right.

where do they all come from?

From what I've seen, there are two major sources, and one slightly less prevalent one:

1. Big companies. If you go through the incubator, you'll find quite a few projects where there was a large code grant, and many initial contributors, from one large company or another. Facebook, IBM, Yahoo, Oracle, etc.

2. Academia. A lot of projects are existing OSS projects being developed as part of academic research, and now the team wants to formalize the project structure and get the project to a wide audience.

The slightly less prevalent one:

3. Just ordinary OSS projects developed $WHEREVER, not necessarily affiliated with any company or university, and the team just decide that Apache would make a good home for their project.

Anyway, I definitely wouldn't call the ASF "a dumping ground for the technology of failed startups". If anything the incubator is designed to prevent that sort of thing. If projects don't successfully complete incubation, they don't become full fledged TLPs. Now, there are people who quibble over how long some projects linger in the incubator, sometimes in a sort of half-dead/half-alive zombie state, but by and large the process works pretty well.

Re: Apache Singa, a Distributed Deep Learning Platform

#18

Post by lazycrazyowl »

ereyes01 wrote: post #9

And Cuban hackers everywhere chuckle at the unfortunate slang cuss word chosen as the name of this project... http://www.urbandictionary.com/define.php?term=Singa

Singh is actually the Indian Sanskrit root word for Lion and is common in south Indian language of Kannada, Tamil and Malayalam which has Simha, Singham and Simham as the word for lion respectively. The Sri Lanka's ethnic majority are called Sinhalese which again is derived from the Indo-Aryan Sinhala language bearing similar influence, meaning the "lion people" or "people with lion blood", while a sword-wielding lion is the central figure on the modern national flag of Sri Lanka.The island nation of Singapore (Singapura) derives its name from the Malay words singa (lion) and pura (city), which in turn is from the Tamil-Sanskrit சிங்க singa सिंह siṃha and पुर புர pura.

Re: Apache Singa, a Distributed Deep Learning Platform

#19

Post by limau »

brianchu wrote: post #10
neel8986 wrote: post #8

Earlier quoted context omitted.

It is in their schedule for december http://singa.incubator.apache.org/develop/schedule.html

It's only single node, multiple GPUs, though, which already exists in other frameworks.

With the parallelism model and abstraction it has to support model/data or hybrid partitioning, and synchronous/asynchronous or hybrid training, it should be easy to extend to GPU cluster. However, training is required only periodically, and if it can be done on existing clusters as efficiently, why not?
Post reply on HN