Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

biorxiv.org

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#41

Post by londons_explore »

tastroder wrote: post #26
lysium wrote: post #16

Earlier quoted context omitted.

Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.

Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.

Most machine learning accelerators have a few non-deterministic operations. The chances that you could run trillions of floating point operations through a GPU and get a bit-for-bit identical result is low.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#42

Post by drb91 »

nl wrote: post #36
drb91 wrote: post #31

Earlier quoted context omitted.

My understanding is that academic authorship credit is political: authors don’t always contribute, and contributors don’t always get credit. Is this not the case?

Not really. Usually the politics goes the other way - people getting an author slot because they are the head of the department or something.

I believe I covered that—authors who don’t contribute.

Also, I know too many angry grad students to believe that contributors always get credit.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#43

Post by tempguy9999 »

londons_explore wrote: post #41
tastroder wrote: post #26

Earlier quoted context omitted.

Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.

Most machine learning accelerators have a few non-deterministic operations. The chances that you could run trillions of floating point operations through a GPU and get a bit-for-bit identical result is low.

Really? I'm not an ML guy so in simple terms, what are these non-deterministic ops? Or are you saying GPUs can be expected to be, basically, faulty?

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#44

Post by p1esk »

nl wrote: post #38
visarga wrote: post #34

Earlier quoted context omitted.

> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the o…

It's not really as clear cut as that. Transformers work well in sequence tasks because both compare well in terms of accuracy but also scale better than a RNNs like a LSTM or a GRU. That means they can be trained on more data. This isn't really the same as CNNs, where they model images by running at different scales. I'm not aware of any cases of Transformers being used particularly successfully on images. They can b…

https://arxiv.org/abs/1904.09925

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#45

Post by londons_explore »

tempguy9999 wrote: post #43
londons_explore wrote: post #41

Earlier quoted context omitted.

Most machine learning accelerators have a few non-deterministic operations. The chances that you could run trillions of floating point operations through a GPU and get a bit-for-bit identical result is low.

Really? I'm not an ML guy so in simple terms, what are these non-deterministic ops? Or are you saying GPUs can be expected to be, basically, faulty?

Both.

Some operations split and join data in non-deterministic ways (especially the order of operations, leading to different floating point rounding). If you shard across multiple machines, weight accumulation order will depend on network latency for example.

Also, GPU's aren't anywhere near as reliable as CPU's when it comes to being able to run for hours without any random bit flips/errors.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#46

Post by nl »

p1esk wrote: post #44
nl wrote: post #38

Earlier quoted context omitted.

It's not really as clear cut as that. Transformers work well in sequence tasks because both compare well in terms of accuracy but also scale better than a RNNs like a LSTM or a GRU. That means they can be trained on more data. This isn't really the same as CNNs, where they model images by running at different scales. I'm not aware of any cases of Transformers being used particularly successfully on images. They can b…

https://arxiv.org/abs/1904.09925

Nice. I guess I'm 8 days behind on the SOTA...

But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system performed slightly worse than the one including the CNN layers.

Also, this isn't really a Transformer architecture, even though it uses Attention.

But maybe this is too much nitpicking? I agree that Attention is a useful primitive - my point is that the Transformer architecture is too specific.

(Also, this is a really nice paper in that it lays out the hyperparameters and training schedules they used. And that Appendix is amazing!)

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#47

Post by jerven »

jostmey wrote: post #13
dnautics wrote: post #9

Earlier quoted context omitted.

> It does a surprisingly good job of predicting protein function across a diverse set of tasks, including ones structural in nature, like the induction of a single neuron that is able, with some degree of accuracy (ρ = 0.33) to distinguish between α helices and β strands (I suspect the network as a whole is far more performant at this task than the single neuron we’ve identified, but we didn’t push this aspect of the…

And predicting protein function is not that hard either. The ground truth labels are often determined by sequence alignment similarity, not by experiment. So the results are far from profound

Doing it right is quite hard. Doing it usefully is even harder [1]. Getting a good training set without to many biases is the really hard part. Generating a ground truth that is actually a truth is very expensive.

I have to read the paper carefully again. But for the contact point prediction I think the training set will cover most of the data used in the validation. Due to they way PDB "sequences" are distributed over UniParc as well as how PDB 3D structures are generated experimentally. i.e. there are 120,000 pdb related sequences in UniParc, but they cover 45,000 ones in UniProtKB. Because PDB derived sequences are rarely full length, often mutated and highly duplicative in coverage.

[1] predicting the root GO terms will give you and insane TP/FP rate but is completely useless.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#48

Post by jerven »

shpongled wrote: post #19
inciampati wrote: post #15

Earlier quoted context omitted.

Bingo. That would be really interesting. And useful. There are probably already enzymes in this data set that have measurements of their behavior. Could this modelling approach be coaxed to find the one with the highest processivity? Or do we need more labeled data?

I'm sure they have a bunch of enzymes in their dataset for which kinetic measurements have been published. Another interesting follow up study would attempting to improve kinetic behavior. They could, for instance, analyze some of the catalytically perfect enzymes out there (TIM, SOD, catalase, etc) and see if the model could project improvements onto existing orthogonal protein classes.

Not in a structured way that is easily useable. Swiss-Prot has most of this data but it is not quite normalized in units. If you did this annoying work I would like to talk to you so we can plug it into Swiss-Prot.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#49

Post by tempguy9999 »

londons_explore wrote: post #45
tempguy9999 wrote: post #43

Earlier quoted context omitted.

Really? I'm not an ML guy so in simple terms, what are these non-deterministic ops? Or are you saying GPUs can be expected to be, basically, faulty?

Both. Some operations split and join data in non-deterministic ways (especially the order of operations, leading to different floating point rounding). If you shard across multiple machines, weight accumulation order will depend on network latency for example. Also, GPU's aren't anywhere near as reliable as CPU's when it comes to being able to run for hours without any random bit flips/errors.

> ...split and join data in non-deterministic ways ... to different floating point rounding

Ah, of course! A very timely reminder, thanks!

> GPU's aren't anywhere near as reliable as CPU's when it comes to being able to run for hours without any random bit flips/errors.

Now that's worrying. A bit flip can't be expected to be skewed towards any particular bit within a float, so it could easily happen in the exponent, skewing a single value by orders of magnitude one way or the other. Combine that with the rest of your 'good' results and yuck. That's very concerning. Thanks for the warning.

Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

#50

Post by return1 »

nl wrote: post #37
return1 wrote: post #21

Earlier quoted context omitted.

and transformers

The Attention Is All You Need paper is where Transforms were introduced: We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Yup, and the state of the art BERT and gpt-2 are both based on transformers.
Post reply on HN