Viewing profile — stephenroller

with hf transformers, you still need to manage GPUs

Why did the share of childless women 35-44 suddenly drop 5 points around 2012? Such a massive difference must be a change in measurement methodology?

The llama1 team did not have a validation set. I don’t know what the Llama2 team did - I left before seeing any of the details. My guess is Llama2 upsamples Wikipedia a good bit, b…

One noteworthy thing is that no one is posting validation curves, only training curves. All these models will happily bring training loss eventually to near zero with infinite comp…

Updating the model happens offline to ensure protections of this. As part of the research on BlenderBot 3 we also have improved detections for safety and can learn from "trolls." h…

We trained on Reddit comments and HackerNews comments.

I run a fairly large open source project ( https://github.com/facebookresearch/ParlAI/ ) and we use mypy. Our experience has been that it can be quite difficult to placate, so we u…

https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327

Support for this was also added to [Fairscale]( https://fairscale.readthedocs.io/en/latest/ ) and [Fairseq]( https://github.com/pytorch/fairseq ) last week. In particular, the Fair…

They don't. But half the people got a placebo and half got the vaccine. 185 in the placebo group got covid, with 30 having severe covid, but 11 with covid and 0 severe in the vacci…

Streets would be much more useful than a true north compass.

Perhaps you are outside the United States? PBS is the public television channel, so perhaps they geolock :(

The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use …

No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another not…

I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still n…

(BlenderBot original developer here) You can read the paper here, which contains much more technical details of how we built it: https://arxiv.org/abs/2004.13637

Hi there, thanks for asking! It's very cool to make research more accessible. Would you be able to rename your experience to "Modified Small BlenderBot without Safety Layer"? And I…

Hi there. One of the original authors of the BlenderBot paper here. This cocohub website is not what we released. It appears that cocohub has removed our safety layer and provided …

Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the comput…

We limited the scope of our study to the PersonaChat dataset, which is a standard benchmark in dialogue. In PersonaChat, both participants are presented with Personas and asked to …

Yes, that's a great response for one of the models to have. :)

We originally set out to see whether we could use conditional generation to sort of adjust the behavior of these neural generative dialogue systems. Much of the original idea was a…

Co-author of the original paper, if you have any questions.