Live data from Hacker News

Show HN: Six Degrees of Wikipedia

sixdegreesofwikipedia.com

331–340 of 354 posts

Re: Show HN: Six Degrees of Wikipedia

#331

After spending way too much time I got 9 degrees of separation. I did piggyback off of someone else's work with "Phinney". https://www.sixdegreesofwikipedia.com/?source=Lion%20Express... I found a ton of 5 degree paths and only a couple of 6 degree ones. Then I pulled out the "big guns" (the dead-end pages category). https://en.wikipedia.org/wiki/Category:Dead-end_pages_from_F...

Damn, I had trouble finding more than 3 degrees

Re: Show HN: Six Degrees of Wikipedia

#332

After spending way too much time I got 9 degrees of separation. I did piggyback off of someone else's work with "Phinney". https://www.sixdegreesofwikipedia.com/?source=Lion%20Express... I found a ton of 5 degree paths and only a couple of 6 degree ones. Then I pulled out the "big guns" (the dead-end pages category). https://en.wikipedia.org/wiki/Category:Dead-end_pages_from_F...

yeah I just cheated by using orphaned articles and managed to 'win'

Re: Show HN: Six Degrees of Wikipedia

#333
post #329
post #13

Creator here. Six Degrees of Wikipedia is a side project I've been sporadically hacking on over the past few years. It was an interesting technical challenge and it's fun to play with the end result. Here's the tech stack: * Frontend: React (Create React App) * Backend: Python Flask * Database: SQLite * Web (frontend) hosting: Firebase Hosting * Server (backend) hosting: Google Compute Engine (it runs fine on a tiny…

How big does the SQLite database get? How do you maintain such great performance? (Other than indexes?)

The resulting SQLite database file is currently 8.3 GB, most of which is taken up by the `links` table. The big performance wins are having a handful of indexes (see the .sql files[1] for the database's schema) and preprocessing a lot of data so I don't have to do duplicate work every time a query occurs. For example, instead of the `links` table going from `source_id` to `target_id` and having a ton of rows which have the same `source_id`, I go from `id` to `outgoing_links` (which is a |-separate string of all source page IDs). Computing each page's incoming and outgoing links is the really heavy work and I only do that at database creation time, using a beefy GCP machine with 8 vCPUs, 52 GB RAM, and a 256 GB SSD. It still takes about an hour, but it's a one time cost and means I can run the actual service on a much smaller machine which won't cost me a fortune to maintain. Also, SQLite is just very fast and performant out of the box, so as usual, it's a matter of choosing the right tools for the job.

[1] https://github.com/jwngr/sdow/tree/master/database

Re: Show HN: Six Degrees of Wikipedia

#335
post #231
post #204

Earlier quoted context omitted.

The downside of bidirectional seems to be that things like place names are linked via short paths through boring articles like "Census designated place" or "City"

The bi-directional nature of the search does not change the end result. It is simply a performance improvement.

[deleted]

Re: Show HN: Six Degrees of Wikipedia

#336
post #13

Creator here. Six Degrees of Wikipedia is a side project I've been sporadically hacking on over the past few years. It was an interesting technical challenge and it's fun to play with the end result. Here's the tech stack: * Frontend: React (Create React App) * Backend: Python Flask * Database: SQLite * Web (frontend) hosting: Firebase Hosting * Server (backend) hosting: Google Compute Engine (it runs fine on a tiny…

Looks pretty cool.

Simple question I have is, are you hitting wikipedia api live? or you have dump of the wikipedia and running through it?

If running through dump do you update it regularly or how?

Thanks in advance.

Re: Show HN: Six Degrees of Wikipedia

#338
post #13

Creator here. Six Degrees of Wikipedia is a side project I've been sporadically hacking on over the past few years. It was an interesting technical challenge and it's fun to play with the end result. Here's the tech stack: * Frontend: React (Create React App) * Backend: Python Flask * Database: SQLite * Web (frontend) hosting: Firebase Hosting * Server (backend) hosting: Google Compute Engine (it runs fine on a tiny…

Looks pretty cool. Simple question I have is, are you hitting wikipedia api live? or you have dump of the wikipedia and running through it? If running through dump do you update it regularly or how? Thanks in advance.

The autocomplete suggestions hit the live Wikipedia API[1]. The actual search algorithm is on a dump of Wikipedia[2], which I plan to update monthly.

[1] https://github.com/jwngr/sdow/blob/f39398d112fecf7b993c64bd4... [2] https://github.com/jwngr/sdow#data-source

Re: Show HN: Six Degrees of Wikipedia

#339
post #337

All I get for every search (no matter what terms) is: "Whoops... something is broken and has been reported. In the mean time, please try a different search."

It was down for a little while due to the traffic but it's back up and running again.

Re: Show HN: Six Degrees of Wikipedia

#340
I like the little trivia that show up, although sometimes they disappear too quickly.

My favorite mentioned which Wikipedia article had the longest article name:

Suzukake no Ki no Michi de "Kimi no Hohoemi o Yume ni Miru" to Itte Shimattara Bokutachi no Kankei wa Dō Kawatte Shimau no ka, Bokunari ni Nannichi ka Kangaeta Ue de no Yaya Kihazukashii Ketsuron no Yō na Mono (https://en.wikipedia.org/wiki/Suzukake_no_Ki_no_Michi_de_%22...)

Post reply on HN