Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking Hacker News

joelgrus.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Hacking Hacker News

#11

Post by sillysaurus3 »

minimaxir wrote: post #8
sillysaurus3 wrote: post #7

Earlier quoted context omitted.

Using that API, you could download 48,000 hacker news stories in 2 days, so if there have been less than 48,000 submissions, then what minimaxir said is true. But first you'd need to generate a list of all 48,000 story ids, and there seems to be no way to actually do that.

No need to generate all IDs beforehand. The search_by_date endpoint is fine. You have to paginate using the created_at_i parameter, not the page parameter. Also, you can set hitsPerPage = 1000. ;)

When trying to access page 2 via that endpoint: http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsP...

"you can only fetch the 1000 hits for this query, contact us to increase the limit"

It was a nice try, but it did seem too good to be true.

Re: Hacking Hacker News

#14

Post by rickdale »

This is totally cool and kudos to you. But be aware there are limits of personalized hacker news. I think Bill Maher put it best when ranting just last night about facebooks customized news feeds:

Newspapers may be old-fashioned, but here's what we're losing if you never see one. They are trying to tell you what's actually important, not just what's important to you. You may not read the whole paper, but you at least see headlines, making you aware that something's going on outside of your microtargeted world of fashion or music or Wiccans or zombies or whatever you're into.

Replace 'newspapers' with hacker news and you get the point.

https://www.youtube.com/watch?v=WohtmZDZCGM

Re: Hacking Hacker News

#15

Post by [deleted] »

minimaxir wrote: post #8
sillysaurus3 wrote: post #7

Earlier quoted context omitted.

Using that API, you could download 48,000 hacker news stories in 2 days, so if there have been less than 48,000 submissions, then what minimaxir said is true. But first you'd need to generate a list of all 48,000 story ids, and there seems to be no way to actually do that.

No need to generate all IDs beforehand. The search_by_date endpoint is fine. You have to paginate using the created_at_i parameter, not the page parameter. Also, you can set hitsPerPage = 1000. ;)

[deleted]

Re: Hacking Hacker News

#18

Post by e15ctr0n »

minimaxir wrote: post #2

FYI, the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit. https://hn.algolia.com/api

There's a list of all the apps that have been built based on this API: http://hn.algolia.com/cool_apps

Re: Hacking Hacker News

#19

Post by minimaxir »

sillysaurus3 wrote: post #11
minimaxir wrote: post #8

Earlier quoted context omitted.

No need to generate all IDs beforehand. The search_by_date endpoint is fine. You have to paginate using the created_at_i parameter, not the page parameter. Also, you can set hitsPerPage = 1000. ;)

When trying to access page 2 via that endpoint: http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsP... "you can only fetch the 1000 hits for this query, contact us to increase the limit" It was a nice try, but it did seem too good to be true.

You can paginate using the created_at_i parameter (edited OP)

Just pass created_at_iI was able to download 500k stories (i.e. about half of HN's 1.26M stories) before I ran into memory issues; I've fixed them and am downloading the rest.

Re: Hacking Hacker News

#20

Post by ThomPete »

rickdale wrote: post #14

This is totally cool and kudos to you. But be aware there are limits of personalized hacker news. I think Bill Maher put it best when ranting just last night about facebooks customized news feeds: Newspapers may be old-fashioned, but here's what we're losing if you never see one. They are trying to tell you what's actually important, not just what's important to you. You may not read the whole paper, but you at least…

Newspapers are trying to show you what they think is actually important. But of course omnibus papers will never really tell you what is important only what is current.
Post reply on HN