Live data from Hacker News

So I made something that's pretty stupid but had fun doing it

hitlerhops.com

51–60 of 148 posts

Re: So I made something that's pretty stupid but had fun doing it

#53
post #7

> but had fun doing it Well, that's more than a lot of people can say about certain projects! Want to give us a run-down of how you put it together?

Definitely, it was actually pretty simple once I figured out what I was going to do. I started from the link graph put together by Henry Haselgrove ( http://users.on.net/~henry/home/wikipedia.htm ) that I found when looking through the EC2 public datasets. I then had a few easy steps. 1) flip the link graph from outgoing to incoming, so from any page I can see what links to it. 2) I found all the distances and paths…

Cool. How long did it take and how big is it?

Also, what's the longest path?

Re: So I made something that's pretty stupid but had fun doing it

#54
post #45
post #41

Earlier quoted context omitted.

Jewish -(redirect)-> Jews -> Adolf Hitler. Both are 3 but December is first in alphabetical so it was the first one recorded.

I find it viscerally painful to see the casual association. 6000 years of culture and all you can think of is some delusional scumbag.

There might be 6000 years of culture, but this app finds degrees of separation(i guess) between wikipedia articles and adolf hitler(that is its purpose). So the association isn't casual and in relation to Adolf Hitler its pretty on topic.

Re: So I made something that's pretty stupid but had fun doing it

#55
Wow, I'm honored by all the feedback, Thanks! Pretty psyched everything is still responsive under the load too.

The dates are pretty lame, luckily looks like Wikipedia removed all the links to dates so when I run against the latest data set they won't be there.

Re: So I made something that's pretty stupid but had fun doing it

#57
post #56

I think you have a bug. Elliptical Curve Cryptography -> AndrU00c3U00a9 Weil -> 1906 -> Adolf Hitler 'AndrU00c3U00a9 Weil' ( http://en.wikipedia.org/wiki/AndrU00c3U00a9_Weil ) doesn't exist. Very cool though.

Thats a encoding bug yeah, supposed to be André Weil. http://en.wikipedia.org/w/index.php?title=Andr%C3%A9_Weil&#3...

Re: So I made something that's pretty stupid but had fun doing it

#58
post #44

I was tempted to say "NSFW in Germany", but then again - that would be provoking havoc. :) Living in Germany I find of find the idea mildy irritating (and the Graphics - erm - far too "comical"), but I love the technical side of the project. Would be interesting to have a way to show the hops between arbitrary two words (but, yeah, the domain name gives away that this is not planned).

That was actually the original idea but the CPU behind it (requires a lot of ram and multiple systems the way I worked it) was too expensive to swing by the wife for a goof off project ;)

You can't pre-compute it with the Floyd-Warshall algorithm in C or something?

EDIT: to try to answer my own calculation on the back of the envelope...

We're going to need n^2 space and n^3 time, where n is the number of nodes. There are about 3.5 million articles (http://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia), and if we assume that the diameter of the Wikipedia graph is 255 or less, we need 1 byte * 3.5 million * 3.5 million = 12.25 TB just to store path lengths for all pairs. It's also going to take a while because locality is going to be hosed. You might be able to do something smarter, like do the all-pairs solutions for something that will fit in memory (top 30000 pages or so), and hope that queries match a long-tail distribution...

Re: So I made something that's pretty stupid but had fun doing it

#59
post #55

Wow, I'm honored by all the feedback, Thanks! Pretty psyched everything is still responsive under the load too. The dates are pretty lame, luckily looks like Wikipedia removed all the links to dates so when I run against the latest data set they won't be there.

Awesome! When will you?
Post reply on HN