Hacker News search

#1

Post by ymn_ayk » Sun, Dec 09, 2012, 5:51 PM UTC

I'm starting to do 'Introduction to CS' from Udacity (http://www.udacity.com/overview/Course/cs101/CourseRev/apr2012). I'm planning to make a Hacker news search app through the course. What do you think, is that a good idea, or not? I know there is something similar; this is my plan: the search will be executed in the domains, referred by the hacker news stories. Thank you

Re: Hacker News search

#2

Post by slash-dot » Sun, Dec 09, 2012, 6:32 PM UTC

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

Re: Hacker News search

#3

Post by dbaupp » Sun, Dec 09, 2012, 6:42 PM UTC

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

There is an unofficial API: http://www.hnsearch.com/api (Provided by the very search engine referred to in the OP, haha!)

Re: Hacker News search

#4

Post by pyre » Sun, Dec 09, 2012, 6:49 PM UTC

dbaupp wrote: post #3

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

There is an unofficial API: http://www.hnsearch.com/api (Provided by the very search engine referred to in the OP, haha!)

Unfortunately there is no API for getting access to personal information on HN (i.e. comments I have made, or stories I've upvoted). You're relegated to scraping if you want that information.

Re: Hacker News search

#5

Post by rikacomet » Sun, Dec 09, 2012, 6:51 PM UTC

speaking of this, there should be one. And a "http://Archive as well.

only a handful of topics make it to the top everyday, but the majority of content is high quality stuff, which if applied at a proper place would help many. What say?

Re: Hacker News search

#6

Post by beatgammit » Sun, Dec 09, 2012, 6:59 PM UTC

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

You can always get around this by throttling your web crawler. It will take a much longer time, but at least you'll be able to read HN in the meantime.

Re: Hacker News search

#7

Post by ymn_ayk » Sun, Dec 09, 2012, 8:06 PM UTC

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

Why will you be banned? Do you know about the reason?

Re: Hacker News search

#8

Post by alphast0rm » Sun, Dec 09, 2012, 8:37 PM UTC

beatgammit wrote: post #6

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

You can always get around this by throttling your web crawler. It will take a much longer time, but at least you'll be able to read HN in the meantime.

The tricky thing when doing this is knowing what rate to stop at without getting permanently banned. I built an Android Market crawler two summers ago, and luckily Google only temp bans (from my experience), so that might be an easier project without any risk.

Re: Hacker News search

#9

Post by jgeralnik » Sun, Dec 09, 2012, 8:50 PM UTC

ymn_ayk wrote: post #7

slash-dot wrote: post #2

Keep in mind that if you crawl this site your ip will be banned. At least mine was when I was playing around with a web crawler i built.

Why will you be banned? Do you know about the reason?

To prevent people from (unintentionally) DDoSing the site.

Re: Hacker News search

#10

Post by arikrak » Sun, Dec 09, 2012, 9:50 PM UTC

I think a practical way to implement simple version would be through Google CSE, but you would have more control if you roll your own search.