All that's missing is a custom Twitter client that does compression upon tweeting and in the various timeline columns ala Tweetdeck, does decompression, it could make for something interesting. Too bad that would be against Twitter's TOS.
A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
11–20 of 39 posts
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#12To extend upon this idea - if you really wanted to maximise the data you could transmit in a single twitter message you could use the full 31bits of unicode (instead of just the chinese subset) and then apply standard lossless data compression techniques to the generated unicode for further improvement.
There are a lot of code points that would need to be filtered out if you do this - Noncharacters, Control codes, High/Low surrogates, Private-Use, Whitespace, and then of course the ones that mutate other code points in the sequence - Bidirectional, Combining characters / diacritical marks. It isn't quite as simple as just combining random 32-bit characters, as I found when creating my URL shortener.
If you want to play around with this, there is a helpful official (but really hard to find) Web app here: http://unicode.org/cldr/utility/properties.html
The filter I ended up using is:
[:Diacritic=No:]&[:Noncharacter_Code_Point=No:]&[:Deprecated=No:]&[:White_Space=No:]&[:General_Category=Math_Symbol:]|[:General_Category=Symbol:]|[:General_Category=Letter:]|[:General_Category=Punctuation:]|[:General_Category=Currency_Symbol:]|[:General_Category=Number:]&[:General_Category!=Modifier_Letter:]&[:General_Category!=Modifier_Symbol:]Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#13Earlier quoted context omitted.
Same basic idea. He's using variable-length input strings and mapping them to unique symbols rather than the other way around, but it seems like he's using entropy measurements to build an optimal trie.
Is an arbitrary bit sequence valid unicode? If so, I'm curious if a simple per-character Huffman encoding of English would be more efficient.
No, it's not. (Although "unicode" is ambiguous, because there are several unicode encodings.) Even if the binary sequence can be decoded as unicode code points, certain combinations of code points aren't allowed (surrogate pairs have to be matched).
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#14I think the most realistic way to compress a tweet would be to replace words like "before" with "b4", "too"/"to" with "2", reduce whitespace (e.g. double spaces to single), and maybe start ripping out vowels ("vowels" -> "vwls").
Although not as efficient as demonstrated above, there are no external dependencies needed; the content can be decompressed by the reader's brain in-place at the slight cost of being difficult to immediately parse/understand.
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#15Would it be possible to use the tweet metadata to cram more data into a tweet?
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#16So Twitter allows 140 (UTF-8?) characters, regardless of the number bytes? The article wasn't clear about this.
Twitters allows 140 codepoints of a sequence in normalization form C.
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#17I think the most realistic way to compress a tweet would be to replace words like "before" with "b4", "too"/"to" with "2", reduce whitespace (e.g. double spaces to single), and maybe start ripping out vowels ("vowels" -> "vwls"). Although not as efficient as demonstrated above, there are no external dependencies needed; the content can be decompressed by the reader's brain in-place at the slight cost of being difficu…
There's also a social status cost, which could be great or small, for example if your readers have an intense dislike for netspeak and poor English.
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#18To extend upon this idea - if you really wanted to maximise the data you could transmit in a single twitter message you could use the full 31bits of unicode (instead of just the chinese subset) and then apply standard lossless data compression techniques to the generated unicode for further improvement.
There are a lot of code points that would need to be filtered out if you do this - Noncharacters, Control codes, High/Low surrogates, Private-Use, Whitespace, and then of course the ones that mutate other code points in the sequence - Bidirectional, Combining characters / diacritical marks. It isn't quite as simple as just combining random 32-bit characters, as I found when creating my URL shortener. If you want to p…
> High/Low surrogates
Surrogates are not codepoints.
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#19Earlier quoted context omitted.
There are a lot of code points that would need to be filtered out if you do this - Noncharacters, Control codes, High/Low surrogates, Private-Use, Whitespace, and then of course the ones that mutate other code points in the sequence - Bidirectional, Combining characters / diacritical marks. It isn't quite as simple as just combining random 32-bit characters, as I found when creating my URL shortener. If you want to p…
> High/Low surrogates Surrogates are not codepoints.
That's not right. Surrogates are code points. That's the whole idea! It means you can express characters beyond the BMP with legacy encodings that were designed back when the entire code space could be coded in 16 bits. Newer encodings like UTF-8 don't have to rely on surrogates to do this because they have that capability at the bytewise encoding level.
Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
#20I think the most realistic way to compress a tweet would be to replace words like "before" with "b4", "too"/"to" with "2", reduce whitespace (e.g. double spaces to single), and maybe start ripping out vowels ("vowels" -> "vwls"). Although not as efficient as demonstrated above, there are no external dependencies needed; the content can be decompressed by the reader's brain in-place at the slight cost of being difficu…
There's also a social status cost, which could be great or small, for example if your readers have an intense dislike for netspeak and poor English.
Here in the UK it's interesting to note that, while text speak was all the rage a few years ago, it faded out.
It was replaceddddd by making wordssss actuallllllly longerrrrrr for no reeeeaaaasonnnnn!!!
Now the text speak is a bit more reined in and not totally incomprehensible like it used to get.