Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

biorxiv.org

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#31

Post by drb91 »

slashcom wrote: post #25
mkolodny wrote: post #22

Earlier quoted context omitted.

I'd guess that wink is hinting that Yann LeCun might've had something to do with this research. Whether that's true or not, I have no idea. (Yann LeCun is a Turing award winner for his work in deep learning)

Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.

My understanding is that academic authorship credit is political: authors don’t always contribute, and contributors don’t always get credit. Is this not the case?

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#32

Post by lysium »

tastroder wrote: post #26
lysium wrote: post #16

Earlier quoted context omitted.

Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.

Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.

You want to be able to set the seed if only you want to be able to debug your program. Pseudo random is sufficient for these models and is independent of any hardware settings. You should not share your random source between concurrent threads, though, but that’s good practice anyway.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#33

Post by visarga »

vanattab wrote: post #17
Havoc wrote: post #14

Earlier quoted context omitted.

>any clear way to reproduce the results. Given that it's evolved I'd imagine this is a given? Or more accurately you could probably duplicate some kind of emergent behaviour but it would be different given different randomized parameters

Is it not possible to use the same seed and random number generator to reproduce the results accurately?

You got to be careful. RNG is being used to initialise the layers but also for mini-batch selection. They are usually different RNG's.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#34

Post by visarga »

ArtWomb wrote: post #11

Fergus Lab at NYU. I believe he's across the hall from Yann LaCunn as well ;) Still a long way from a Theory of Biogenesis. But a good next step is using a differentiable model to predict novel proteins which have no analogue in Nature. Much like Materials Genome researchers searching for stable phases of matter! "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the sa…

> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet

The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the one in computer vision from a few years ago.

What is needed besides spatial translation invariance (CNN) and temporal invariance (LSTM) is permutation invariance. Whenever the problem can be described as a graph, then the ordering of the vertices and edges should not matter. You can't do that with CNNs and LSTMs, but you can do it with Graph neural nets and Transformers.

Apparently Transformers are the best for language modelling (GPT-2), playing games (Dota2 from OpenAI), composing music and possibly now in modelling proteins. I assume they will play a huge role in working with graph structured data, with multiple entities and relations.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#35

Post by nl »

tastroder wrote: post #26
lysium wrote: post #16

Earlier quoted context omitted.

Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.

Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.

For the most part, yes.

There are specific CUDA operations which are not guaranteed to be reproducible though, as well as some CuDNN operations which are non-determanistic without performance sacrifice, and this does cause real problems.

See https://pytorch.org/docs/stable/notes/randomness.html for some reasonable docs on this.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#36

Post by nl »

drb91 wrote: post #31
slashcom wrote: post #25

Earlier quoted context omitted.

Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.

My understanding is that academic authorship credit is political: authors don’t always contribute, and contributors don’t always get credit. Is this not the case?

Not really. Usually the politics goes the other way - people getting an author slot because they are the head of the department or something.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#37

Post by nl »

return1 wrote: post #21
mfatica wrote: post #3

Earlier quoted context omitted.

I would say you also need a fair bit of data too...

and transformers

The Attention Is All You Need paper is where Transforms were introduced:

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#38

Post by nl »

visarga wrote: post #34
ArtWomb wrote: post #11

Fergus Lab at NYU. I believe he's across the hall from Yann LaCunn as well ;) Still a long way from a Theory of Biogenesis. But a good next step is using a differentiable model to predict novel proteins which have no analogue in Nature. Much like Materials Genome researchers searching for stable phases of matter! "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the sa…

> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the o…

It's not really as clear cut as that.

Transformers work well in sequence tasks because both compare well in terms of accuracy but also scale better than a RNNs like a LSTM or a GRU. That means they can be trained on more data.

This isn't really the same as CNNs, where they model images by running at different scales. I'm not aware of any cases of Transformers being used particularly successfully on images.

They can be used on graphs of course, by translating the problem into a graph walk problem (ala DeepWalk).

All the examples you gave (language modelling, Dota2, music and protein modelling) are setup as sequence prediction problems, so are perfect for Transformers.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#39

Post by seppel »

tastroder wrote: post #26
lysium wrote: post #16

Earlier quoted context omitted.

Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.

Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.

There are many CS conferences where you can/should submit a VM image to reproduce the results. See, e.g.: http://cavconference.org/2018/artifact-submission-and-evalua...
Post reply on HN