Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: my weekend project – browse jobs posted to HN

hackernewsjob.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Show HN: my weekend project – browse jobs posted to HN

#21

Post by typpo »

also_on_sunday wrote: post #15
SanjayUttam wrote: post #13

Just curious what technologies you are using? Also how quickly will this update on the 1st when a new thread is posted...again - just curious. Great job! Much better than ctrl-f-ing my way through postings [just like you were doing].

Thanks. To answer your first question: I'm using a custom scraper to get the results. I request the job pages through a random http proxy every so often, parse them, and update the cached json of all the comments. Getting a reliable, free, machine-friendly list of http proxies is harder than I thought. It was automatic, but the proxy provider I was using went down (doh!) so now I manually pull one from a list and tri…

Why is it necessary to use a proxy?

Re: Show HN: my weekend project – browse jobs posted to HN

#22

Post by also_on_sunday »

typpo wrote: post #21
also_on_sunday wrote: post #15

Earlier quoted context omitted.

Thanks. To answer your first question: I'm using a custom scraper to get the results. I request the job pages through a random http proxy every so often, parse them, and update the cached json of all the comments. Getting a reliable, free, machine-friendly list of http proxies is harder than I thought. It was automatic, but the proxy provider I was using went down (doh!) so now I manually pull one from a list and tri…

Why is it necessary to use a proxy?

hacker news black listed the ip address of my server after scraping 1 post's comments about once every fifteen minutes for a day. The best way I could think to get around this was an http proxy.

Re: Show HN: my weekend project – browse jobs posted to HN

#26

Post by cj »

habosa wrote: post #25

Can I ask how you parsed job postings from the rest? I have a few programs that would greatly benefit from the ability to distinguish the jobs from normal posts. It doesn't seem that they are different in html...

I presume he's not parsing and instead simply manually entering the Who's Hiring threads and crawling those. If you look down the list you'll see a few entries that are not job posts.

Edit: And the YC job postings are located at http://news.ycombinator.com/jobs

Re: Show HN: my weekend project – browse jobs posted to HN

#27

Post by dionidium »

habosa wrote: post #25

Can I ask how you parsed job postings from the rest? I have a few programs that would greatly benefit from the ability to distinguish the jobs from normal posts. It doesn't seem that they are different in html...

Job postings don't have an associated comment count

Re: Show HN: my weekend project – browse jobs posted to HN

#28

Post by also_on_sunday »

cj wrote: post #26
habosa wrote: post #25

Can I ask how you parsed job postings from the rest? I have a few programs that would greatly benefit from the ability to distinguish the jobs from normal posts. It doesn't seem that they are different in html...

I presume he's not parsing and instead simply manually entering the Who's Hiring threads and crawling those. If you look down the list you'll see a few entries that are not job posts. Edit: And the YC job postings are located at http://news.ycombinator.com/jobs

Correct - I'm just parsing all the comments and only showing the "top level" ones. I use this library I wrote: https://github.com/brianc/node-hacker-news-parser which builds a tree structure of the comments.

Re: Show HN: my weekend project – browse jobs posted to HN

#29

Post by etcet »

also_on_sunday wrote: post #22
typpo wrote: post #21

Earlier quoted context omitted.

Why is it necessary to use a proxy?

hacker news black listed the ip address of my server after scraping 1 post's comments about once every fifteen minutes for a day. The best way I could think to get around this was an http proxy.

You can use this link http://news.ycombinator.com/unban?ip= ip> to get your IP unbanned. I think it only works once though.

Re: Show HN: my weekend project – browse jobs posted to HN

#30

Post by shurane »

also_on_sunday wrote: post #15
SanjayUttam wrote: post #13

Just curious what technologies you are using? Also how quickly will this update on the 1st when a new thread is posted...again - just curious. Great job! Much better than ctrl-f-ing my way through postings [just like you were doing].

Thanks. To answer your first question: I'm using a custom scraper to get the results. I request the job pages through a random http proxy every so often, parse them, and update the cached json of all the comments. Getting a reliable, free, machine-friendly list of http proxies is harder than I thought. It was automatic, but the proxy provider I was using went down (doh!) so now I manually pull one from a list and tri…

Very awesome. Got any more projects up your sleeve?

Is it better to have an xml feed and load that instead of a JSON file? I wrote a scraper that dumped university courses at my school to a JSON file. I would load that JSON file and load the elements to a table. Mine was pretty dang slow.

But your site holds up fine.

Here's my site if you're at all interested: http://lo.leet.la/jola/bootstrap/docs/sunny.html#

Relvant JS code: http://lo.leet.la/jola/bootstrap/docs/assets/js/sunny.js

Post reply on HN