Label yo axis son.
Expanding on this could lead to some interesting insights. How about "could of" and "could have"? and the old "could care less"...
Show HN: Automated writing analysis of Twitter and Reddit feeds
11–20 of 20 posts
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#12Label yo axis son. Expanding on this could lead to some interesting insights. How about "could of" and "could have"? and the old "could care less"...
Those might work for Twitter, but there may be some lurking variables on Reddit. While they never bother to catch yea and anyways, Redditers do have a tendency to pile on about the few grammatical/spelling errors they can spot. For example, if a post title says "could of", there's likely going to be multiple comments discussing the error and thus tripping my hooks.
Edit: As for the axes, I made some stylistic decisions that I wouldn't dare to make were this being submitted to the Annual New Media Socio-Linguistics Journal.
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#13Quibble: I don't think I'd consider either "yea" or "anyways" misspellings per se, at least in informal chatting. "Anyways" is heard in some spoken dialects, so you'd expect it in written forms of the same dialects. And I think of "yea" (in modern usage) as a variant of "yeah", like "yah" is. All of them are phonetic spellings of dialectal versions of "yes" to begin with (along with "yep" and "yup" and such). They do…
I chose those two to start things off for two reasons: The first is that they stand out to me as pretty basic misspellings that don't appear in any sort of popular literature, so it speaks to someone's reading experience. You can take that for whatever importance it is to you, personally. The second reason is their frequency. If the unit of measurement was larger than a half-hour or hour, I could measure a bunch of o…
I guess I'm disagreeing that they're "basic misspellings" in the first place, or related to someone's reading experience. Informal spoken English is often different from written English, and some people Tweet closer to how they'd speak. That doesn't mean that the same people would write the same way if they were writing a book (they probably wouldn't use "yeah" in a book, either), or that a certain dialect of spoken English is "incorrect".
I think you're measuring something other than spelling here, closer to measuring prevalence of certain dialects. Especially in the "anyways" example: if there's something objectionable about that word, it's a usage objection, not a spelling objection. People really do say "anyways", and that would be the correct way of writing it if you accept that usage. Spelling it "enyways" or "annyways" or something would be an orthographic error, on the other hand.
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#14Earlier quoted context omitted.
This is a cool idea. Did you account for the smaller amount of users at night vs day? It might be even more interesting to do comparisons between subreddits and comparisons between hashtags or followers of certain figures. Also do you have a link to the source? Keep up the good stuff!
Thanks. It compares misspellings to correct spelling counterparts (anyways:anyway), so that, in itself, should account for sample size changes in Twitter. The Reddit sample size (should) stay constant, as I'm grabbing /all/comments every 15 seconds, which from everything I've seen is about 3 times longer than it takes for it to turn over. I have been thinking about subreddits and hashtags, but I'm not sure what would…
Maybe you could do a one time analysis of different subreddits rather than a time series analysis?
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#15I made this over the last two days in Node.js. The analysis is still pretty simple, and I'd like to expand it over time. It updates every 30 minutes, and you can already see that there's some significant shifts in literacy between the daytime and very early morning. I had originally started the project to grab Flesch-Kincaid grade levels, but it was taking me 45 minutes to analyze about 20 pages of comments, so I swi…
I like the general idea and presentation, but if you're seriously desiring to measure social shift, I think basing that judgment on "yea/yeah" and "anyway/anyways" usage is problematic. The former is dropping a silent letter from an arbitrarily-spelled informal word. The latter is a difference in dialect (i.e. culture), not intelligence.
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#16Earlier quoted context omitted.
I chose those two to start things off for two reasons: The first is that they stand out to me as pretty basic misspellings that don't appear in any sort of popular literature, so it speaks to someone's reading experience. You can take that for whatever importance it is to you, personally. The second reason is their frequency. If the unit of measurement was larger than a half-hour or hour, I could measure a bunch of o…
I guess I'm disagreeing that they're "basic misspellings" in the first place, or related to someone's reading experience. Informal spoken English is often different from written English, and some people Tweet closer to how they'd speak. That doesn't mean that the same people would write the same way if they were writing a book (they probably wouldn't use "yeah" in a book, either), or that a certain dialect of spoken…
There's no link between spoken and written English in the context of yea:yeah, as they're pronounced the same but either written correctly or incorrectly.
As for anyways, I think your classification there will depend on who you speak to (which is the point of the analysis).
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#17Quibble: I don't think I'd consider either "yea" or "anyways" misspellings per se, at least in informal chatting. "Anyways" is heard in some spoken dialects, so you'd expect it in written forms of the same dialects. And I think of "yea" (in modern usage) as a variant of "yeah", like "yah" is. All of them are phonetic spellings of dialectal versions of "yes" to begin with (along with "yep" and "yup" and such). They do…
I chose those two to start things off for two reasons: The first is that they stand out to me as pretty basic misspellings that don't appear in any sort of popular literature, so it speaks to someone's reading experience. You can take that for whatever importance it is to you, personally. The second reason is their frequency. If the unit of measurement was larger than a half-hour or hour, I could measure a bunch of o…
The problem with your conclusion that 'anyways' is a misspelling and you chose it because it is popular (high 'frequency') is that it displays a common misunderstanding purists have of a language's evolution.
As Jorge Luis Borges once said, all words were once neologisms.
For something to be standardized or make it into a dictionary, lexicographers look at its popularity. If we reduce language to the most simplest idea, it is exactly this: a set of popular phonetic sounds from which humans derive agreed similar meaning.
The phenomenon of adding an extra sound to the end of a word is called paragoge.
For anyone that is interested and is not familiar with it, linguistics has a lot to say on the addition and subtraction of sounds at the beginning, middle, and end of words. Here is a link: http://en.wikipedia.org/wiki/Metaplasm
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#18"Yea" pronounced "yay" is a valid word with practically the same meaning as "yeah" (affirmative). Granted "yea" is archaic and most of the instances you are catching are just yeahs with the "h" dropped, but it would be nice to choose something that is definitively wrong. The anyway/anyways is a better choice and I hope that "anyways" does not become widely accepted because it is like Bieber to my ears.
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#19Earlier quoted context omitted.
I chose those two to start things off for two reasons: The first is that they stand out to me as pretty basic misspellings that don't appear in any sort of popular literature, so it speaks to someone's reading experience. You can take that for whatever importance it is to you, personally. The second reason is their frequency. If the unit of measurement was larger than a half-hour or hour, I could measure a bunch of o…
The problem with your conclusion that 'anyways' is a misspelling and you chose it because it is popular (high 'frequency') is that it displays a common misunderstanding purists have of a language's evolution. As Jorge Luis Borges once said, all words were once neologisms. For something to be standardized or make it into a dictionary, lexicographers look at its popularity. If we reduce language to the most simplest id…
Great points. I went into this knowing full-well that to many, many people, these misspellings were not considered as such. As the graph suggests, though, the majority of people still use the "older" spellings of "yeah" and "anyway". There are two ways of spelling each one (with "yea" and "anyways" considered to be the more nascent), and I'm approaching this from the standpoint of why the spellings are changing. Is the language evolving (suggesting improvement) or devolving (suggesting perpetuated errors). We could discuss linguistics, but what indicators are there from the data showing Reddit has a lower percentage of yeas and anywayss compared to Twitter? Is it simply the medium? How does that data compare to your anecdotal classification of the respective aggregate intelligence level of both sites' users? What do you think the data would say about HN vs. Youtube on the frequency of these (as you would consider) both valid spellings? This all boils down to me labeling the graph as common misspellings, but ignoring that, you should be able to make of the graphs what you will, as there's no associated hypothesis or conclusion.
Re: Show HN: Automated writing analysis of Twitter and Reddit feeds
#20"Yea" pronounced "yay" is a valid word with practically the same meaning as "yeah" (affirmative). Granted "yea" is archaic and most of the instances you are catching are just yeahs with the "h" dropped, but it would be nice to choose something that is definitively wrong. The anyway/anyways is a better choice and I hope that "anyways" does not become widely accepted because it is like Bieber to my ears.
I'm working on the assumption of statistical significance of people using "yea" pronounced "yeah" vs. the very few people who might have reason to take to Twitter to use "yea" in a voting or biblical context.