SpaCy PyTorch Transformers

#1

Post by syllogism » Fri, Aug 02, 2019, 8:23 PM UTC

SpaCy PyTorch Transformers
explosion.ai

Re: SpaCy PyTorch Transformers

#2

Post by clmnt » Fri, Aug 02, 2019, 9:09 PM UTC

Very cool!

Re: SpaCy PyTorch Transformers

#3

Post by 3wolf » Fri, Aug 02, 2019, 10:55 PM UTC

I'm surprised transfer learning via fine-tuning large transformer models hasn't taken off more in the public consciousness a la image recognition models. In my experience, the results can be staggering with very small amounts of training data.

Re: SpaCy PyTorch Transformers

#4

Post by shiredude95 » Fri, Aug 02, 2019, 11:13 PM UTC

Awesome! Cant wait to try this out. Are there any plans in the future to incorporate sentence/word generation?

Re: SpaCy PyTorch Transformers

#5

Post by syllogism » Fri, Aug 02, 2019, 11:15 PM UTC

shiredude95 wrote: post #4

Awesome! Cant wait to try this out. Are there any plans in the future to incorporate sentence/word generation?

That would be cool, yeah. We haven't had anything like that in spaCy so far, but the results from these models can be very good. I think with a good API it could be very useful.

Re: SpaCy PyTorch Transformers

#6

Post by arathore » Fri, Aug 02, 2019, 11:20 PM UTC

Spacy also has one of the best documentations that I have come across. I really like the use of Tufte-style notes for code examples. https://spacy.io/api

Re: SpaCy PyTorch Transformers

#7

Post by ipsum2 » Fri, Aug 02, 2019, 11:58 PM UTC

3wolf wrote: post #3

I'm surprised transfer learning via fine-tuning large transformer models hasn't taken off more in the public consciousness a la image recognition models. In my experience, the results can be staggering with very small amounts of training data.

Google (and I assume most ML providers) offers transfer learning on image models: https://cloud.google.com/vision/automl/docs/

Re: SpaCy PyTorch Transformers

#8

Post by ma2rten » Sat, Aug 03, 2019, 12:34 AM UTC

3wolf wrote: post #3

I'm surprised transfer learning via fine-tuning large transformer models hasn't taken off more in the public consciousness a la image recognition models. In my experience, the results can be staggering with very small amounts of training data.

It's a very new thing. Pretrained ImageNet models were first released around 2011. Pretrained transformer models have only been released recently.

Also, transformers are only useful for short text, not full documents (AFAIK).

Re: SpaCy PyTorch Transformers

#9

Post by goldemerald » Sat, Aug 03, 2019, 12:43 AM UTC

This stuff is great! Will example training files be uploaded for GLUE/SQAUD/etc like on the HuggingFace's implementation?

Re: SpaCy PyTorch Transformers

#10

Post by jonbaer » Sat, Aug 03, 2019, 12:45 AM UTC

arathore wrote: post #6

Spacy also has one of the best documentations that I have come across. I really like the use of Tufte-style notes for code examples. https://spacy.io/api

Also the recent SpaCy IRL talks are well worth watching @ https://www.youtube.com/watch?v=hNPwRPg9BrQ&list=PLBmcuObd5A...