Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A.I. Doesn't Get Black Twitter

inverse.com

Post reply on HN
Print view View original

61–70 of 105 posts

Re: A.I. Doesn't Get Black Twitter

#61

Post by briholt »

xiaoma wrote: post #3

"Approximately 8 of the 319 million people in the United States read the Wall Street Journal, about 2 percent of the population. If you look at the language — standardized English — being fed into many natural language processing units, it’s based on the language of that 2 percent. " It's hard to take an article seriously when it opens with a logical fallacy. Yes, the WSJ uses standard English and only 8 million peop…

> When researchers evaluated their model against natural language processing tools...researchers found that the software flagged African-American English as “not English” at a much higher frequency than standard English.

Likewise, notice the deceptive switch between "English" and "standard English." So basically language bots trained to identify standard English are correctly identifying non-standard English dialects as not standard English.

Re: A.I. Doesn't Get Black Twitter

#63

Post by morgante »

thaumasiotes wrote: post #62
morgante wrote: post #55

Earlier quoted context omitted.

What are the "formal rules" of AAVE?

https://en.wikipedia.org/wiki/African_American_Vernacular_En...

I'm looking for formal rules. I'm not saying AAVE doesn't exist, I'm saying its rules (such as they are) are not formalized. Tons of rule books exist for Standard English and part of learning it is to memorize the correct rules. I don't believe any formal equivalent exists for AAVE.

Re: A.I. Doesn't Get Black Twitter

#64

Post by thaumasiotes »

ralfd wrote: post #21
a3n wrote: post #16

Earlier quoted context omitted.

There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."

> I expect to almost totally lose touch with the speech of "these kids today." Is this long term really a problem though? Regression to the mean: No one cares about the specific slang of the 60s anymore. And the hip slang of today will be out of fashion in 15 years. In a similar vain the extreme T9 keyboard text message abbreviations of the youth in the 2000s did go out of fashion because of automatic spell checking…

> Is this long term really a problem though?

How long term? What's "a problem"? We know with certainty that given the passage of enough time, the speech of "these kids today" will be completely unintelligible to you.

Re: A.I. Doesn't Get Black Twitter

#65

Post by Dylan16807 »

gugagore wrote: post #37
a3n wrote: post #16

Earlier quoted context omitted.

There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."

Do you think written Standard English is less ambiguous than any other English, or written language? Because I don't see why that would be the case.

>Do you think written Standard English is less ambiguous than any other English

Definitely. Things are clearer when you write formally and strictly adhere to grammatical rules. When I'm speaking casually I omit words and (ab)use punctuation much differently, and while that is perfectly acceptable to do, it is more complex and ambiguous to parse.

As far as I'm aware the same pattern exists in a lot of languages and dialects, and it's possible to define formal AAVE too.

Re: A.I. Doesn't Get Black Twitter

#66

Post by iamdave »

morgante wrote: post #63
thaumasiotes wrote: post #62

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/African_American_Vernacular_En...

I'm looking for formal rules. I'm not saying AAVE doesn't exist, I'm saying its rules (such as they are) are not formalized. Tons of rule books exist for Standard English and part of learning it is to memorize the correct rules. I don't believe any formal equivalent exists for AAVE.

What, then, are your parameters for formal? It has to be in a book, codified and signed off on? If so, aren't these kind of antithetical to AAVE on the face of it?

Re: A.I. Doesn't Get Black Twitter

#67

Post by morgante »

iamdave wrote: post #66
morgante wrote: post #63

Earlier quoted context omitted.

I'm looking for formal rules. I'm not saying AAVE doesn't exist, I'm saying its rules (such as they are) are not formalized. Tons of rule books exist for Standard English and part of learning it is to memorize the correct rules. I don't believe any formal equivalent exists for AAVE.

What, then, are your parameters for formal? It has to be in a book, codified and signed off on? If so, aren't these kind of antithetical to AAVE on the face of it?

> It has to be in a book, codified and signed off on?

The point is for rules to be formal then the must be formalized somehow and codified. This could be online or in a book, but the point is that there must be some clear delineation between when the rules are followed and when they are broken. Otherwise what's the meaning of "formal" rules?

> If so, aren't these kind of antithetical to AAVE on the face of it?

Yes. My argument is that AAVE, almost by definition, is an informal dialect without formalized rules.

This is getting rather off topic, but I think it relates to the original point of why NLP might start with Standard English even if you are not biased. A large corpus of Standard English text (such as from the WSJ) will generally be very internally consistent precisely because it follows a set of formal rules codified into a style guide. As there is no such equivalent for AAVE, even gathering a large and internally consistent corpus of AAVE text seems prohibitively difficult. That being said, I do hope researchers are working on gathering text from Twitter to build up new training sets.

Re: A.I. Doesn't Get Black Twitter

#68

Post by morgante »

gugagore wrote: post #37
a3n wrote: post #16

Earlier quoted context omitted.

There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."

Do you think written Standard English is less ambiguous than any other English, or written language? Because I don't see why that would be the case.

> Do you think written Standard English is less ambiguous than any other English, or written language?

Absolutely. It follows a strict set of formal rules and you can readily find sources to learn those rules. Heck, copyeditors regularly debate the specific nuances of how to consistently write Standard English.

Other dialects (and spoken language) are far less formalized. Where is the AP Stylebook or Strunk & White for AAVE?

Re: A.I. Doesn't Get Black Twitter

#69

Post by ubernostrum »

morgante wrote: post #67
iamdave wrote: post #66

Earlier quoted context omitted.

What, then, are your parameters for formal? It has to be in a book, codified and signed off on? If so, aren't these kind of antithetical to AAVE on the face of it?

> It has to be in a book, codified and signed off on? The point is for rules to be formal then the must be formalized somehow and codified. This could be online or in a book, but the point is that there must be some clear delineation between when the rules are followed and when they are broken . Otherwise what's the meaning of "formal" rules? > If so, aren't these kind of antithetical to AAVE on the face of it? Yes.…

The point is for rules to be formal then the must be formalized somehow and codified

So every form of English is an "informal dialect" then? Because this ain't French with the Académie publishing strict rules for use of the language. Do you also say that the languages of remote Amazon tribes aren't "real languages" because they don't have a formal government body publishing written rules?

Or do you just want to bash on AAVE and are grasping at straws for reasons why?

Re: A.I. Doesn't Get Black Twitter

#70

Post by morgante »

ubernostrum wrote: post #69
morgante wrote: post #67

Earlier quoted context omitted.

> It has to be in a book, codified and signed off on? The point is for rules to be formal then the must be formalized somehow and codified. This could be online or in a book, but the point is that there must be some clear delineation between when the rules are followed and when they are broken . Otherwise what's the meaning of "formal" rules? > If so, aren't these kind of antithetical to AAVE on the face of it? Yes.…

The point is for rules to be formal then the must be formalized somehow and codified So every form of English is an "informal dialect" then? Because this ain't French with the Académie publishing strict rules for use of the language. Do you also say that the languages of remote Amazon tribes aren't "real languages" because they don't have a formal government body publishing written rules? Or do you just want to bash…

> So every form of English is an "informal dialect" then?

Yes, the majority of spoken English does not follow the rules of Standard English. Pretending that such rules don't exist is willful ignorance though: the WSJ obviously write a more formalized version of English than teenagers do in text messages.

> Do you also say that the languages of remote Amazon tribes aren't "real languages" because they don't have a formal government body publishing written rules?

Nowhere did I say that AAVE is "not a real language" because it's less formalized than Standard English. Prior to spelling reforms, English itself was extremely inconsistent and informal, but I certainly don't pretend that it wasn't a language.

> Or do you just want to bash on AAVE and are grasping at straws for reasons why?

I'm not trying to bash AAVE. In fact, I'd even posit that the reason AAVE isn't more codified is perhaps because of racial bias which treated it simply as "incorrect" English instead of a separate dialect worthy of formalization. Pretending that all languages are equally formalized is simply willful ignorance though.

Post reply on HN