A multimodal dataset with one trillion tokens
1–10 of 55 posts
Re: A multimodal dataset with one trillion tokens
#2[dead]
Re: A multimodal dataset with one trillion tokens
#3[dead]
Re: A multimodal dataset with one trillion tokens
#4More info on the Salesforce blog:
https://blog.salesforceairesearch.com/mint-1t/
Re: A multimodal dataset with one trillion tokens
#5How effective would modeling raw byte sequences be, with the individual bytes as the "tokens", and a vocabulary of 256 elements?
You could then train on any kind of digital data.
Re: A multimodal dataset with one trillion tokens
#6More info on the Salesforce blog: https://blog.salesforceairesearch.com/mint-1t/
Wow did not expect sales force to be behind this.
It’s basically free advertising for technical people to join sales force.
Re: A multimodal dataset with one trillion tokens
#7More info on the Salesforce blog: https://blog.salesforceairesearch.com/mint-1t/
Wow did not expect sales force to be behind this. It’s basically free advertising for technical people to join sales force.
Salesforce has long been involved in publishing quality NLP papers, especially during Stephen Merity's tenure.
Smerity's papers are some of my favourite. Check out
https://ar5iv.labs.arxiv.org/html/1708.02182
And my all-time favourite
Re: A multimodal dataset with one trillion tokens
#8How effective would modeling raw byte sequences be, with the individual bytes as the "tokens", and a vocabulary of 256 elements? You could then train on any kind of digital data.
Forget bytes, go for bits. Vocab of size 2. At a theoretical level all of AI comes down to a classifier that is able to predict the next bit given a string of bits. Check out Tsetlin Machines. At some point we will be doing it in hardware.
Re: A multimodal dataset with one trillion tokens
#9How effective would modeling raw byte sequences be, with the individual bytes as the "tokens", and a vocabulary of 256 elements? You could then train on any kind of digital data.
Somewhat inefficient for text, very inefficient for images, specially if you work in pixel space. The max context a model today has been trained is 1M tokens, which takes up a lot of memory. Even if context was not an issue, to generate a 1000x1000 image would take ~3 hours on 100token/s inference.
Google has trained an encoder/decoder LLM on bytes called ByT5[1]
Re: A multimodal dataset with one trillion tokens
#10How effective would modeling raw byte sequences be, with the individual bytes as the "tokens", and a vocabulary of 256 elements? You could then train on any kind of digital data.
You might be interested in reading up on DNA sequence llm models and tooling.