Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A multimodal dataset with one trillion tokens

github.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: A multimodal dataset with one trillion tokens

#21

Post by nighthawk454 »

jszymborski wrote: post #7
j7ake wrote: post #6

Earlier quoted context omitted.

Wow did not expect sales force to be behind this. It’s basically free advertising for technical people to join sales force.

Salesforce has long been involved in publishing quality NLP papers, especially during Stephen Merity's tenure. Smerity's papers are some of my favourite. Check out https://ar5iv.labs.arxiv.org/html/1708.02182 And my all-time favourite https://ar5iv.labs.arxiv.org/html/1911.11423

Hey thanks for those Smerity links, hadn't run across his work yet, second one in particular looks great

Re: A multimodal dataset with one trillion tokens

#22

Post by ks2048 »

It looks like it contains data from CommonCrawl and ArXiv. It's not clear what kind of processing they did, but sometimes these releases seem like just repackaging existing datasets with your name own name on them. It's not hard to get bulk downloads from these sources directly.

I thought CommonCrawl truncated files at 1MB. I wonder if the PDFs for CommonCrawl were re-fetched from the URLs. That could be useful if they provide simple way to get those full files.

Re: A multimodal dataset with one trillion tokens

#23

Post by anas-awadalla »

ks2048 wrote: post #22

It looks like it contains data from CommonCrawl and ArXiv. It's not clear what kind of processing they did, but sometimes these releases seem like just repackaging existing datasets with your name own name on them. It's not hard to get bulk downloads from these sources directly. I thought CommonCrawl truncated files at 1MB. I wonder if the PDFs for CommonCrawl were re-fetched from the URLs. That could be useful if th…

Hello! Creator of MINT here.

We do a lot of pre-processing of commoncrawl (which in its raw form isn’t all that useful for training models). This includes heuristics to remove low quality text and images and deduplicating documents, paragraphs, and images. All of these are crucial to achieve good training performance.

On your point regarding PDFs, we actually don’t constraint ourselves to the 1MB files and do our own downloading of PDFs!

Re: A multimodal dataset with one trillion tokens

#24

Post by anas-awadalla »

sva_ wrote: post #20

Does it make sense to measure a dataset in tokens? Shouldn't it be tokenizer-agnostic? I.e. the OpenAI tokenizer encodes about ~4 characters per token, but I could also have a tokenizer that does 1 character per token leading to a ~4x increase in token count (relative to the OpenAI tokenizer.)

Hello! Totally agree that tokens will be model dependent. We chose to calculate tokens using the GPT-2 tokenizer as that is a common metric used by other datasets like fineweb. So this should roughly give you a sense of how large the data is in comparison to others. We report other metrics too like number of documents and number of images.

Re: A multimodal dataset with one trillion tokens

#25

Post by gotaran »

j7ake wrote: post #6
punnerud wrote: post #4

More info on the Salesforce blog: https://blog.salesforceairesearch.com/mint-1t/

Wow did not expect sales force to be behind this. It’s basically free advertising for technical people to join sales force.

I’m skeptical of the caliber of talent at Salesforce given the unusable state of their core product.

Re: A multimodal dataset with one trillion tokens

#26

Post by paxys »

gotaran wrote: post #25
j7ake wrote: post #6

Earlier quoted context omitted.

Wow did not expect sales force to be behind this. It’s basically free advertising for technical people to join sales force.

I’m skeptical of the caliber of talent at Salesforce given the unusable state of their core product.

The people building CRM software aren't also the ones doing AI research. The two have nothing to do with each other.

Re: A multimodal dataset with one trillion tokens

#27

Post by paxys »

dpifke wrote: post #17
brianjking wrote: post #15

What's the license though?

From https://huggingface.co/datasets/mlfoundations/MINT-1T-HTML#l... : We release MINT-1T under a CC-BY-4.0 license, designating it primarily as a research artifact. While the dataset is freely available, users are responsible for ensuring its legal use in commercial settings. Users must independently verify compliance with applicable laws before employing MINT-1T for commercial purposes. Same page includes this cave…

Ah yes, the "if you get busted for copyright violations it's not our problem" license.

Re: A multimodal dataset with one trillion tokens

#28

Post by ks2048 »

anas-awadalla wrote: post #23
ks2048 wrote: post #22

It looks like it contains data from CommonCrawl and ArXiv. It's not clear what kind of processing they did, but sometimes these releases seem like just repackaging existing datasets with your name own name on them. It's not hard to get bulk downloads from these sources directly. I thought CommonCrawl truncated files at 1MB. I wonder if the PDFs for CommonCrawl were re-fetched from the URLs. That could be useful if th…

Hello! Creator of MINT here. We do a lot of pre-processing of commoncrawl (which in its raw form isn’t all that useful for training models). This includes heuristics to remove low quality text and images and deduplicating documents, paragraphs, and images. All of these are crucial to achieve good training performance. On your point regarding PDFs, we actually don’t constraint ourselves to the 1MB files and do our own…

I see. Thanks for the reply. I opened one of the tar files and see now how it has extracted the text into json files.

Re: A multimodal dataset with one trillion tokens

#29

Post by gotaran »

paxys wrote: post #26
gotaran wrote: post #25

Earlier quoted context omitted.

I’m skeptical of the caliber of talent at Salesforce given the unusable state of their core product.

The people building CRM software aren't also the ones doing AI research. The two have nothing to do with each other.

The ones doing AI research would be working at more prestigious institutions.

Re: A multimodal dataset with one trillion tokens

#30

Post by ericjmorey »

gotaran wrote: post #29
paxys wrote: post #26

Earlier quoted context omitted.

The people building CRM software aren't also the ones doing AI research. The two have nothing to do with each other.

The ones doing AI research would be working at more prestigious institutions.

Why make such a ridiculous assumption?
Post reply on HN