Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning has a size problem

heartbeat.fritz.ai

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Deep learning has a size problem

#31

Post by Izkata »

joe_the_user wrote: post #30
blt wrote: post #15

IMO this is not a problem. The people building insanely huge models are expanding the set of tasks that can be done by a computer. Who cares how much memory it takes? Historically, computationally expensive methods eventually become cheap. In the 1980's, researchers had access to Crays to develop physics model, graphics, etc. requiring lots of floating point math and memory. Meanwhile, for the home computers, game pr…

IMO this is not a problem. The people building insanely huge models are expanding the set of tasks that can be done by a computer. Who cares how much memory it takes? But are they? The example in the article describes an incremental improvement in a benchmark in exchange for a massive increasing in training time. Deep learning has achieved success on a number of tasks that previously computers had been unable to do.…

> And if it is true that deep learning is stuck on just expanding what it's already doing, it might be the fundamental next advance might come from one person with one machine rather than a massive team with a massive machine. Consider that neural nets as a theory had been around since the 1990s if not the 1960s but the fundamental advantage of DL came when grad students could use GPU in the 2010s, not when massively parallel machines came into existence (quite a bit earlier).

One thing that I can't help wondering, however sci-fi it sounds, is if model simplifications like in this post might lead to models humans can fully understand, which then might lead to new styles of traditional programing - opening up whole new ways of doing things.

Re: Deep learning has a size problem

#32

Post by bitL »

turnersr wrote: post #25
bitL wrote: post #23

Earlier quoted context omitted.

BERT is already outdated, but still useful as you need only 1 Titan RTX to retrain its BERT_large model via transfer learning.

What methods make BERT outdated? Do you have pointers to other options?

e.g. XLNet:

https://arxiv.org/abs/1906.08237

Re: Deep learning has a size problem

#34

Post by galkk »

I never understand such remarcs

> Given the power requirements per card, a back of the envelope estimate put the amount of energy used to train this model at over 3X the yearly energy consumption of the average American.

So what? Training model is the hardest part, then you just reuse results

> First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to make every application and device better, we won’t get there with massive models that take large amounts of time and money to train.

So what? I can't run weather simulation on my laptop.

Re: Deep learning has a size problem

#35

Post by hnaccy »

ivalm wrote: post #16
buboard wrote: post #11

The article starts with NLP models and then mentions the successes of increasingly smaller vision models. NLP seems to be an outlier in increasingly becoming a pissing contest. The models are too big and not particularly useful. openAI spread FUD about their model but after their release , it's rather underwhelming. Yeah you can output some text that's readable and paraphrasing reddit, but what about understanding ,…

As someone who was able to generate a model for production based on BERT that outperformed all our previous attempts, I have to say transformers really are a game changer. They are not the end all be all, but they are really, really good as being the basis of many different classification tasks.

Any tips in terms of taking BERT style model to production?

Re: Deep learning has a size problem

#36

Post by sgt101 »

galkk wrote: post #34

I never understand such remarcs > Given the power requirements per card, a back of the envelope estimate put the amount of energy used to train this model at over 3X the yearly energy consumption of the average American. So what? Training model is the hardest part, then you just reuse results > First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to ma…

What are the applications of deep learning that look like weather simulations (as in one run -> results to 10m people?) In my experience deep learning systems are aimed at applications that are single use 1 run -> 1 person.

The training cost is more important than you think as well. To train a model normally requires 10's or 100's of experiments, meaning that we are consuming 30 -> 3k people's carbon, and the application of the model is typically narrow, so we end up doing 4 or 5 projects per year per group... meaning that we could spend 10's of k carbon per team to produce $10m's benefit. I wonder if we can justify this at all?

Re: Deep learning has a size problem

#38

Post by chongli »

galkk wrote: post #34

I never understand such remarcs > Given the power requirements per card, a back of the envelope estimate put the amount of energy used to train this model at over 3X the yearly energy consumption of the average American. So what? Training model is the hardest part, then you just reuse results > First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to ma…

So what? Training model is the hardest part, then you just reuse results

I doubt anyone is going to want to run a 33GB model on their phone.

So what? I can't run weather simulation on my laptop.

You only need to run the weather simulation once and then broadcast your forecast to everyone’s devices. You can’t do that with NLP. In order to be useful, NLP models need to run on different input data for every user. With a giant 33GB model, that means round-tripping to the data centre.

If you have to run everything in the cloud, your applications are limited. The cost is also very high, given that there are way more user devices than servers in the world. That means you need to build more data centres if you plan to run these giant models for every application you want to offer your users.

Re: Deep learning has a size problem

#39

Post by phoboslab »

chongli wrote: post #38
galkk wrote: post #34

I never understand such remarcs > Given the power requirements per card, a back of the envelope estimate put the amount of energy used to train this model at over 3X the yearly energy consumption of the average American. So what? Training model is the hardest part, then you just reuse results > First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to ma…

So what? Training model is the hardest part, then you just reuse results I doubt anyone is going to want to run a 33GB model on their phone. So what? I can't run weather simulation on my laptop. You only need to run the weather simulation once and then broadcast your forecast to everyone’s devices. You can’t do that with NLP. In order to be useful, NLP models need to run on different input data for every user. With a…

> I doubt anyone is going to want to run a 33GB model on their phone.

Why not? Many modern phones have upwards of 512GB of storage. 33 GB for a useful model seems entirely reasonable to me.

Re: Deep learning has a size problem

#40

Post by RocketSyntax »

pheug wrote: post #18
RocketSyntax wrote: post #5

Deep learning doesn't parallelize well. Would be cool if you could loan CPU cycles on your phone or home computers while at work.

Actually it parallelizes extremely well, so that large companies are able to create monster models like mentioned in the article in the first place by just throwing money at the problem with TPUs and similar highly parallelized accelerators. It just doesn't lend itself well to distributed computing due to e.g. throughput requirements.

That's just vertical scale. Distributed is what I was referring to. See comment below.
Post reply on HN