Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Draft of the Fast.ai Book

github.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Draft of the Fast.ai Book

#41

Post by mnks »

mci wrote: post #33
jph00 wrote: post #20

Earlier quoted context omitted.

Yes you're right, in our NLP course we used Turkish as our example. But for the book I mentioned Polish due to this paper: https://arxiv.org/abs/1810.10222 . But as you say, now the word "agglutinative" isn't technically correct. I'm actually not sure what the right word is to describe languages that have lots of big compounds with no spaces. (Which is the key issue here, as to why we need subword tokenization techni…

The term you are looking for may be "highly inflected". Side note: IMHO, you are exaggerating the ability of Polish to form long compounds. Dissecting the "Bezbarwne zielone idee wściekle śpią" example from https://arxiv.org/pdf/1810.10222.pdf#page=3 reveals no words longer than 4 morphemes: bez-BARW-n-e ZIEL-on-e IDE-e WŚCIEK-l-e ŚP-ią, where I put word roots in uppercase and bound morphemes in lowercase. The longes…

It's not about the number of letters in the compounds, but about the number of morphemes.

Your "powychodziłybyście" example could be translated as "you (feminine, plural) would have been going out". With the word tokenization, you get (ignoring comma and brackets) 8 tokens in English and one token in Polish. Now you can have three persons, two genders, two numbers, an imperfective or perfective verb, etc. resulting in combinatorial growth of word tokens in Polish. If you have all word forms for "go out" and you want to add "go in", in English you would add a single token "in", and in Polish you add all the tokens with "-wy-" replaced by "-w-". As a result in Polish you end up with much bigger vocabulary. Additionally you need bigger training corpus as you cannot learn the tokens independently. For example, if you know the meaning of "he ate" and "she wrote", you should be able to guess the meaning of "he wrote", as you've seen all of the tokens. In Polish it's "Zjadł", "Napisała" and "Napisał" - all of the word tokens are different.

Using the subword tokenization instead of word-level tokenization is kind of similar to using a normalized database instead of unnormalized one. It's not about one form being more complex than the other as they're equivalent. After all, will written English be much more complex if we remove all whitespaces? :)

Re: Draft of the Fast.ai Book

#42

Post by mci »

mnks wrote: post #41
mci wrote: post #33

Earlier quoted context omitted.

The term you are looking for may be "highly inflected". Side note: IMHO, you are exaggerating the ability of Polish to form long compounds. Dissecting the "Bezbarwne zielone idee wściekle śpią" example from https://arxiv.org/pdf/1810.10222.pdf#page=3 reveals no words longer than 4 morphemes: bez-BARW-n-e ZIEL-on-e IDE-e WŚCIEK-l-e ŚP-ią, where I put word roots in uppercase and bound morphemes in lowercase. The longes…

It's not about the number of letters in the compounds, but about the number of morphemes. Your "powychodziłybyście" example could be translated as "you (feminine, plural) would have been going out". With the word tokenization, you get (ignoring comma and brackets) 8 tokens in English and one token in Polish. Now you can have three persons, two genders, two numbers, an imperfective or perfective verb, etc. resulting i…

I agree with what you wrote. I did not object to subword tokenization that let you(?) win the competition. I objected to GP's assertion that one can add many morphemes together to create very long "words" in Polish, which made casual readers think of stringing morphemes like German compounds while the number of morphemes in Polish words is bounded by 7, maybe by 8.

Re: Draft of the Fast.ai Book

#44

Post by he11ow »

This is not intending to minimize in the slightest the amazing work that Jeremy does - I am a huge fan.

But Fast.ai has TWO co-founders, and somehow, Rachel doesn't seem to get any credit in these discussions (not the book specifically, I'm talking about the overall enterprise). Not quite sure why; A lot of the content on the website is written by her, and it's clear she adds a lot of value to the endeavor as a whole.

Re: Draft of the Fast.ai Book

#45

Post by sytelus »

bonoboTP wrote: post #38
sytelus wrote: post #27

Earlier quoted context omitted.

Wait... so if you buy book then it seizes to be GPLed? This is quite confusing. For DL research, most code is MIT licenced and legal folks at many industrial labs would be quite hesitent to permit use of code from this repo with feels like legal minefield with different restrictions spread over multiple places including LICENSE, README, fastai website and perhaps printed book. I would highly recommand converting to o…

I don't get this perspective about the GPL. Look, they are giving you something for free, including the source code and the right to build upon it and publish modified versions. You can do basically whatever you want with it, as long as you pass on the freedoms that were granted to you. Is that unfair? Enjoying getting freedoms but not passing them on is not nice.

I get GPL and fully appreciate its philosophy. The problem happens when you actually use it in practice. Because of its viral nature, anyone with different licensing must convert to GPL if they use your code. For many scenarios, this is actually not possible not just because of commercial secrets but the potential for opening up for security vulnerabilities when you don’t have resources or competitions where you should keep code secret until some time or simply because you have dependencies on other code which is very expensive to get rid off. Due to this reason, many companies forbid the use of GPL licensed software as well as release anything under it (because then you can’t use your own code!). Many other companies simply don't want the headache of checking all of their mess of legacy codebases with a myriad of dependencies that would be hard to untangle into GPL compatible open-source release. The legal and economic overhead when you use or release GPLed code is non-trivial. For this reason, the vast majority of open-source code released by big tech companies on GitHub is MIT/BSD licensed, which ironically is more "freeier" than GPL.

Re: Draft of the Fast.ai Book

#46

Post by ratsimihah »

jph00 wrote: post #21
ratsimihah wrote: post #6

It looks promising! Minor point, a requirements.txt file or something would be convenient to get started quickly.

Once the book is released there will be a whole website and prebuilt environments and lots more to get started quickly. We didn't expect the draft to get this much attention, frankly!

That's exciting! If pull requests are enabled I can always send one once I'm up and running! Looking forward to checking out the good stuff in there!

Re: Draft of the Fast.ai Book

#47

Post by kriro »

sytelus wrote: post #17
pritovido wrote: post #3

It always amazes me how bad some technical people is at basic promotion: What is Fastai? Why do I need it? Something as basic as an elevator speech that introduces your product in your github page and book intro can mean 10x or 100x more sales. If you force people into having to search it for you, you have already lost most of them. For this author it is as you already know everything about Fastai, but if you did, yo…

You are downvoted by fanboys but you are exactly right. I am surrounded by researchers working in DL and I have say at least 40% of them have never heard of FastAI or Jeremy Howard. However folks who are active on Twitter, listening to popular podcasts, popular media, HN etc would be very familiar with name Jeremy Howard and what FastAI is and need no introduction. In research world, an astonishing number of good res…

I guess it depends a bit which field they work in exactly. I'd be rather surprised if rigorous DL researchers in NLP haven't heard of him because I expect "Universal language model fine-tuning for text classification" (and tbh. also "Fine-tuned language models for text classification" due to the universality of the idea) to show up in any half-decent literature review of the field.

Most DL researchers I know also have a pretty good knowledge of available libraries and make it a habit to check them pretty often.

Re: Draft of the Fast.ai Book

#48

Post by phaker »

jph00 wrote: post #20
mci wrote: post #4

> There are also "agglutinative languages", like Polish, which can add many morphemes together to create very long "words" which include a lot of separate pieces of information. [1] Polish does not work this way. Source: I am Polish. Perhaps jph00 meant Turkish. Issue filed. [1] https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb

Yes you're right, in our NLP course we used Turkish as our example. But for the book I mentioned Polish due to this paper: https://arxiv.org/abs/1810.10222 . But as you say, now the word "agglutinative" isn't technically correct. I'm actually not sure what the right word is to describe languages that have lots of big compounds with no spaces. (Which is the key issue here, as to why we need subword tokenization techni…

After reading that section of the book i think the language property you're after is 'highly synthetic':

https://en.wikipedia.org/wiki/Synthetic_language

There's a spectrum between synthetic and analytic languages ( https://en.wikipedia.org/wiki/Synthetic_language#Synthetic_a... ) and those closer to the synthetic end are the ones giving you trouble.

Polish will be subtype of synthetic called fusional/inflected which means things need to be adjusted to fit together, agglutinative languages are those that use mainly agglutination where morphemes are stuck together as is:

https://en.wikipedia.org/wiki/Agglutinative_language

Since it's a spectrum / categorization based on features, all languages will show these features to various degrees. E.g. the famous "anti|dis|establish|ment|ari|an|ism" in english and "anty|samo|u|bez|przedmiot|owia|nie" as a similar example in polish (both from https://pl.wikipedia.org/wiki/Aglutynacyjno%C5%9B%C4%87 ), or more humble "houseboat" or "bitwise".

There are also polysynthetic languages, which is the name for the extreme of this spectrum, but there are no familiar examples of these (Mayan languages, Ainu, Inuit, Aleut are only i recognize from those mentioned on wikipedia).

Re: Draft of the Fast.ai Book

#49

Post by FranzFerdiNaN »

he11ow wrote: post #44

This is not intending to minimize in the slightest the amazing work that Jeremy does - I am a huge fan. But Fast.ai has TWO co-founders, and somehow, Rachel doesn't seem to get any credit in these discussions (not the book specifically, I'm talking about the overall enterprise). Not quite sure why; A lot of the content on the website is written by her, and it's clear she adds a lot of value to the endeavor as a whole…

I can think of a reason she isn’t getting the credit she deserves.

Re: Draft of the Fast.ai Book

#50

Post by jph00 »

he11ow wrote: post #44

This is not intending to minimize in the slightest the amazing work that Jeremy does - I am a huge fan. But Fast.ai has TWO co-founders, and somehow, Rachel doesn't seem to get any credit in these discussions (not the book specifically, I'm talking about the overall enterprise). Not quite sure why; A lot of the content on the website is written by her, and it's clear she adds a lot of value to the endeavor as a whole…

Thank you for mentioning Rachel! :) She is working as the Founding Director of the Center for Applied Data Ethics nowadays, which is a very full-time job. So she hasn't been involved much in fastai v2 or the book (other than chapter 3, of which she's a co-author).

She created and taught the NLP and Computational Linear Algebra courses, and has written most of the material on the fast.ai blog, and of course (as noted) co-founded fast.ai. Overall, I'd agree that she doesn't get as much credit as she deserves. That's perhaps partly due to her increasing focus on ethics issues, which aren't generally discussed much on HN (sadly).

I also would say that Sylvain Gugger doesn't get as much credit as he should -- he has been an equal partner with me in creating the book and fastai library.

(I discussed this response with Rachel prior to posting it.)

Post reply on HN