A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#1

Post by Umalu » Sat, Sep 03, 2011, 2:37 PM UTC

A Tweet is Worth (at Least) 140 Words With this Compression Algorithm
thevirtuosi.blogspot.com

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#2

Post by bmalicoat » Sat, Sep 03, 2011, 4:59 PM UTC

Good article, but isn't the author describing Huffman codes?

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#3

Post by waffle_ss » Sat, Sep 03, 2011, 5:02 PM UTC

Unicode is really fun. In a similar vein, my first Rails app was a URL shortener that also takes advantage of Twitter's Unicode character counting method:

http://menosgrande.org

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#4

Post by RodgerTheGreat » Sat, Sep 03, 2011, 5:23 PM UTC

bmalicoat wrote: post #2

Good article, but isn't the author describing Huffman codes?

Same basic idea. He's using variable-length input strings and mapping them to unique symbols rather than the other way around, but it seems like he's using entropy measurements to build an optimal trie.

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#5

Post by jconnop » Sat, Sep 03, 2011, 6:15 PM UTC

To extend upon this idea - if you really wanted to maximise the data you could transmit in a single twitter message you could use the full 31bits of unicode (instead of just the chinese subset) and then apply standard lossless data compression techniques to the generated unicode for further improvement.

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#6

Post by blauwbilgorgel » Sat, Sep 03, 2011, 6:16 PM UTC

Great hacking. If you like this topic, this should be relevant: http://stackoverflow.com/questions/891643/twitter-image-enco... (Compressing images in tweets)

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#7

Post by cpeterso » Sat, Sep 03, 2011, 6:25 PM UTC

So Twitter allows 140 (UTF-8?) characters, regardless of the number bytes? The article wasn't clear about this.

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#8

Post by waffle_ss » Sat, Sep 03, 2011, 6:47 PM UTC

cpeterso wrote: post #7

So Twitter allows 140 (UTF-8?) characters, regardless of the number bytes? The article wasn't clear about this.

Here's the Twitter dev article that I used for designing my URL shortener: https://dev.twitter.com/docs/counting-characters#Twitter_Cha...

Basically, they use the Normalization Form C of Unicode normalization which counts code points, not UTF-8 bytes.

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#9

Post by akavi » Sat, Sep 03, 2011, 6:49 PM UTC

RodgerTheGreat wrote: post #4

bmalicoat wrote: post #2

Good article, but isn't the author describing Huffman codes?

Same basic idea. He's using variable-length input strings and mapping them to unique symbols rather than the other way around, but it seems like he's using entropy measurements to build an optimal trie.

Is an arbitrary bit sequence valid unicode? If so, I'm curious if a simple per-character Huffman encoding of English would be more efficient.

Re: A Tweet is Worth (at Least) 140 Words With this Compression Algorithm

#10

Post by waitwhat » Sat, Sep 03, 2011, 6:51 PM UTC

A hack for the sake of doing a hack.

If you really want to compress as many words as you want into a tweet, just include a link: http://www.example.com/really-long-article.txt