BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
1–8 of 8 posts
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#2Fantastic!
> The code and pre-trained model will be available at https://goo.gl/language/bert. Will be released before the end of October 2018.
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#3"We demonstrate the importance of bidirectional
pre-training for language representations". can some one help me understand what bidirectional and pre-trained means?
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#4"We demonstrate the importance of bidirectional pre-training for language representations". can some one help me understand what bidirectional and pre-trained means?
* bidirectional - build representations of the current word by looking into both the future and the past
* pre-trained - train on lots of language modelling data (e.g. billions of words of wikipedia) and then train on the task you really care about but starting from the parameters learnt from the language modelling task.
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#5Fantastic! > The code and pre-trained model will be available at https://goo.gl/language/bert . Will be released before the end of October 2018.
[deleted]
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#6Fantastic! > The code and pre-trained model will be available at https://goo.gl/language/bert . Will be released before the end of October 2018.
eager for the incoming code-)
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#7Is this comparable to fast.ai's ULMfit, which also promises an unsupervised pretrained model, that can be tuned to best state-of-the-art NLP tasks?
Re: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
#8Is this comparable to fast.ai's ULMfit, which also promises an unsupervised pretrained model, that can be tuned to best state-of-the-art NLP tasks?
The big picture is similar. But ULMfit uses amd-lstm for the language modeling, bert uses masked LM instead. Bert has some other tricks like sentence prediction as well.