Viewing profile — craffel

(different author, not Stella) To your first question: Unpublished experiments done by the BigScience architecture and scaling WG suggest that training on book corpus yields a boos…

(author here) The paper/model/code was just made public today. This may be why no one is talking about it yet. Regarding whether the size is a hassle: It's possible to run inferenc…

Yes, that's me! Sorry if I'm being overeager, I like talking about my research!

Agreed! The interesting thing is that basic unsupervised pre-training seems to produce a model which functions not only as a knowledge base but also an NLU system which can effecti…

It's slightly more than that -- it also involves lowercasing and removing articles before testing for string equality.

We include MASS in our empirical survey (see e.g. section 3.3.2 of our paper, https://arxiv.org/pdf/1910.10683.pdf ). FWIW, people were pre-training Transformers before MASS, e.g. …

The blogpost has a summary of our paper from October (a bit late, sorry!) but also has some (fun?) new results on closed-book question answering and fill-in-the-blank text generati…

Thanks, fixed!

Yes, unfortunately we have to rely on the very brittle "exact match" method of evaluating whether an answer is correct. FWIW and perhaps surprisingly, this is the primary way quest…

It actually can be more pernicious than that: https://arxiv.org/abs/1802.08232 However note that the dataset used to train GPT-2 is about 20x smaller than C4. I'm not 100% sure how…

To be clear, here is the claim we make in the paper (we did not write the title of this post to HN): > For SuperGLUE, we improved upon the state-of-the-art by a large margin (from …

Hi, one of the paper authors here. Indeed this is a good question. A couple of comments: - Common Crawl overall is a sparse web dump, it is unlikely that the month we used includes…

Hi, one of the paper's authors here. We didn't submit our model's predictions for the AX-b task yet, we just copied over the predictions from the example submission. We will submit…