Language, music, and now amino acid sequences. Attention is all you need.
I would say you also need a fair bit of data too...
and transformers
Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
21–30 of 51 posts
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#22Fergus Lab at NYU. I believe he's across the hall from Yann LaCunn as well ;) Still a long way from a Theory of Biogenesis. But a good next step is using a differentiable model to predict novel proteins which have no analogue in Nature. Much like Materials Genome researchers searching for stable phases of matter! "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the sa…
> I believe he's across the hall from Yann LaCunn as well ;) I'm having a hard time processing what the wink might possibly mean in this context. No sarcasm intended.
I'd guess that wink is hinting that Yann LeCun might've had something to do with this research. Whether that's true or not, I have no idea.
(Yann LeCun is a Turing award winner for his work in deep learning)
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#23> The resulting model maps raw sequences to representations of biological properties without labels or prior domain knowledge.
A couple of questions:
1. What are those representations?
2. Also what is "biological function"?
3. What kind of information does the learned representation extract that is not already in the "biological properties" it is trained to map to?
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#24I find this paper to be so steeped in hype and dogma so as to be nearly incomprehensible.
Which is a shame, because it's a reasonable approach. I just wish they just frickin described what they did instead of spending the whole paper monologuing and showcasing unconvincing experiments. No need to justify what you're doing, just do it.
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#25Earlier quoted context omitted.
> I believe he's across the hall from Yann LaCunn as well ;) I'm having a hard time processing what the wink might possibly mean in this context. No sarcasm intended.
I'd guess that wink is hinting that Yann LeCun might've had something to do with this research. Whether that's true or not, I have no idea. (Yann LeCun is a Turing award winner for his work in deep learning)
Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#26Earlier quoted context omitted.
>any clear way to reproduce the results. Given that it's evolved I'd imagine this is a given? Or more accurately you could probably duplicate some kind of emergent behaviour but it would be different given different randomized parameters
Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.
Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#27Earlier quoted context omitted.
Usually you use an RNG for which you can publish the seed. So, although it’s random, you can reproduce the results.
Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.
reproducibility should be something that's baked into an experiment's design.
so, if their experiment was designed such that reproduction is inherently difficult, they should have designed it in a better way, and they should've used a toolset that wouldn't run into that problem.
a non-reproducible experiment isn't necessarily completely without value, but it's a thing that everyone should look askance at till it proves its worth.
(apologies if my comments don't apply to this experiment and if it is reproducible -- i didn't have time to read through the OP, but i thought this reply was still a worthwhile response to its specific parent comment)
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#28Earlier quoted context omitted.
Glancing through the paper it seems like they use the recent Transformer model. Does whatever underlying stack they use expose something to share RNG seeds and the exact hardware optimizations your environment applies during training? Otherwise "publishing the seed" sounds nice but might not be as trivial as the phrase suggests.
reproducibility should be something that's baked into an experiment's design. so, if their experiment was designed such that reproduction is inherently difficult, they should have designed it in a better way, and they should've used a toolset that wouldn't run into that problem. a non-reproducible experiment isn't necessarily completely without value, but it's a thing that everyone should look askance at till it prov…
No that's absolutely a fair and true point, my comment was more pointed at the RNG aspect. I have not looked into this specific one either but normally people would hopefully not publish their best randomly achieved run if the system cannot reproduce it or similar results.
That being said the paper in question doesn't seem to reference open source code anyway so I guess my point was kind of moot, apologies.
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#29I find these emergent behaviours fascinating: https://youtu.be/gaFKqOBTj9w
FPGA do interesting things when allowed to exploit sidechannels/analog effects: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50....
Re: Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences
#30Like some of the other ML/AI posts that made it to the top page today, this research too does not give any clear way to reproduce the results. I looked through the pre-print page as well as the full manuscript itself. Without reproducibility and transparency in the code and data, the impact of this research is ultimately limited. No one else can recreate, iterate, and refine the results, nor can anyone rigorously eva…
[deleted]