Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Word Tree in D3.js

jasondavies.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Show HN: Word Tree in D3.js

#11

Post by jasondavies »

jQueryIsAwesome wrote: post #8

Doesn't work in Chrome 25 in W7 XMLHttpRequest cannot load http://www-958.ibm.com/software/data/cognos/manyeyes/datasets/alice-in-wonderland-by-lewis-carroll/versions/1.txt. Origin http://www.jasondavies.com is not allowed by Access-Control-Allow-Origin. Failed to load resource: the server responded with a status of 504 (Gateway Time-out) http://www.jasondavies.com/xhr?url=http%3A%2F%2Fwww-958.ibm.com%2Fsoftware%2Fda…

Try now? I was previously loading some datasets directly from Many Eyes, but I'm using my own copies now to relieve load on their servers.

Re: Show HN: Word Tree in D3.js

#13

Post by jasondavies »

michaldudek wrote: post #12

This is super awesome! Any plans on releasing this on github?

Yes. I will probably add the core wordtree layout as a plugin to https://github.com/d3/d3-plugins

The whole application ties together text processing, data retrieval, the wordtree and longscroll.js for fast rendering of the text view on the right-hand side: https://github.com/d3/d3-plugins/tree/master/longscroll

Re: Show HN: Word Tree in D3.js

#15

Post by jasondavies »

njx wrote: post #3

beautiful, it needs to filter stop words

I tend to agree with Wattenberg and Viégas that it's interesting to treat all words and punctuation equally, but it's certainly a matter of opinion and it would be simple enough to tokenise the input data differently.

Re: Show HN: Word Tree in D3.js

#17

Post by j4ke »

jasondavies wrote: post #13
michaldudek wrote: post #12

This is super awesome! Any plans on releasing this on github?

Yes. I will probably add the core wordtree layout as a plugin to https://github.com/d3/d3-plugins The whole application ties together text processing, data retrieval, the wordtree and longscroll.js for fast rendering of the text view on the right-hand side: https://github.com/d3/d3-plugins/tree/master/longscroll

Any way to shift the text processing to the client? I'd like to use the bookmarklet on some academic papers (many behind a paywall) and the few I've tried only seem to parse the abstract...I assume this is because the text processing is happening server-side, but I could be wrong.

Alternatively, could you release your backend code as well? I'd like to run this on larger corpora.

Very elegant and useful project!

Re: Show HN: Word Tree in D3.js

#18

Post by jasondavies »

j4ke wrote: post #17
jasondavies wrote: post #13

Earlier quoted context omitted.

Yes. I will probably add the core wordtree layout as a plugin to https://github.com/d3/d3-plugins The whole application ties together text processing, data retrieval, the wordtree and longscroll.js for fast rendering of the text view on the right-hand side: https://github.com/d3/d3-plugins/tree/master/longscroll

Any way to shift the text processing to the client? I'd like to use the bookmarklet on some academic papers (many behind a paywall) and the few I've tried only seem to parse the abstract...I assume this is because the text processing is happening server-side, but I could be wrong. Alternatively, could you release your backend code as well? I'd like to run this on larger corpora. Very elegant and useful project!

The text is in fact processed in the client, and is quite fast even for large corpora such as the whole Bible: http://www.jasondavies.com/wordtree/?source=kjv.txt&pref...

It attempts to access URLs directly but this only works if the server sends the appropriate CORS headers (hardly ever).

Otherwise, it falls back to using a proxy, which means the client only sees what the proxy sees. However, you can also paste raw text on the main page.

I could imagine modifying the bookmarklet so it lifts the text directly from the browser instead of just copying the URL. This would solve the proxy issue neatly and would also work for local-only or intranet sites, for which the proxy also fails.

Re: Show HN: Word Tree in D3.js

#19

Post by pjet »

jasondavies wrote: post #15
njx wrote: post #3

beautiful, it needs to filter stop words

I tend to agree with Wattenberg and Viégas that it's interesting to treat all words and punctuation equally, but it's certainly a matter of opinion and it would be simple enough to tokenise the input data differently.

Yes, I saw that reference after posting it here. It makes sense. I also see it is understanding combination of 2 and 3 words together, very brilliant!

Recently I have dabbled into d3 and used your site for lot of inspiration. I created something very similar to analyze text from web pages but using bubbles

[ http://bit.ly/WFTxf5 ]

Re: Show HN: Word Tree in D3.js

#20

Post by Vivtek »

jasondavies wrote: post #11
jQueryIsAwesome wrote: post #8

Doesn't work in Chrome 25 in W7 XMLHttpRequest cannot load http://www-958.ibm.com/software/data/cognos/manyeyes/datasets/alice-in-wonderland-by-lewis-carroll/versions/1.txt. Origin http://www.jasondavies.com is not allowed by Access-Control-Allow-Origin. Failed to load resource: the server responded with a status of 504 (Gateway Time-out) http://www.jasondavies.com/xhr?url=http%3A%2F%2Fwww-958.ibm.com%2Fsoftware%2Fda…

Try now? I was previously loading some datasets directly from Many Eyes, but I'm using my own copies now to relieve load on their servers.

Works fine, same setup. Nice work.
Post reply on HN