Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: News Searching API

datanews.io

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Show HN: News Searching API

#11

Post by caballeto »

asdkhadsj wrote: post #4

Do you host the content as well? What's the legality of that? I ask because I want to embed some archiving and "reader mode" logic into an app of mine that would be FOSS and self hosted. However that means each individual would be effectively scraping and archiving, and possibly p2p spreading, news content (as data sources) . So I'm curious if there is some underlying "fair use"-like mechanism that allows Archive, Ou…

This is a great question. Thus far I thought, that the content that is publicly available without any limitation (e.g. membership access), can be scraped by anyone. You can take a look at hiQ Labs vs. LinkedIn [1]. LinkedIn's public data was scraped by data analytics company, the ruling was against LinkedIn.

I also think that the use case matters, I don't republish their content on the site, but merely provide it via API. Technically, it could be argued, that they could get this data themselves, but it is easier for them to use a service similar to this one to simplify things.

[1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Re: Show HN: News Searching API

#12

Post by caballeto »

jslakro wrote: post #8

How do you get news from its sources?

First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.

Re: Show HN: News Searching API

#13

Post by jwhiz22 »

bvm wrote: post #5
caballeto wrote: post #2

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-…

How are you licensing the content?

He's not.

Re: Show HN: News Searching API

#14

Post by ibdf »

Tried it.. pretty straight forward. Free account rate is good for personal use. I think it comes down to 4 calls per day which is reasonable if you are using this as maybe your daily news fetcher.

Re: Show HN: News Searching API

#15

Post by caballeto »

jwhiz22 wrote: post #13
bvm wrote: post #5

Earlier quoted context omitted.

How are you licensing the content?

He's not.

Right, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.

Re: Show HN: News Searching API

#16

Post by caballeto »

ibdf wrote: post #14

Tried it.. pretty straight forward. Free account rate is good for personal use. I think it comes down to 4 calls per day which is reasonable if you are using this as maybe your daily news fetcher.

Thanks for the feedback! Concerning rate limiting, I think I may have placed rules a bit too strict, as the system is hosted on AWS Free Tier. There is definitely a room for increase in RPS per user, but I wanted to play safe.

Re: Show HN: News Searching API

#17

Post by overcast »

caballeto wrote: post #2

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-…

The prices are high because you’re paying for licensing costs.

Re: Show HN: News Searching API

#18

Post by sbr464 »

caballeto wrote: post #7
sbr464 wrote: post #3

Earlier quoted context omitted.

Nice work. Will share feedback soon after testing.

Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.

Seemed pretty straightforward after a few tests. The only thing I noticed was regarding unique identifiers, the URL in the sources endpoint didn't seem to exactly match the URL attached to a headline? I could be wrong though, and not a big issue. Was more thinking about data cleanup and linking data easily (article/sources/authors etc).

Re: Show HN: News Searching API

#19

Post by caballeto »

sbr464 wrote: post #18
caballeto wrote: post #7

Earlier quoted context omitted.

Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.

Seemed pretty straightforward after a few tests. The only thing I noticed was regarding unique identifiers, the URL in the sources endpoint didn't seem to exactly match the URL attached to a headline? I could be wrong though, and not a big issue. Was more thinking about data cleanup and linking data easily (article/sources/authors etc).

Thanks very much for the feedback. Yes, I think your point is right. For some domains, I try to match the url with sources from "sources" list, and for others I extract source field directly from the url, I think there is a small bug when doing the former, thanks for pointing out. I will fix it soon.

Re: Show HN: News Searching API

#20

Post by artembugara »

Disclaimer: I am a co-founder of a similar News API service https://newscatcherapi.com/

No news API solution returns the full body text of the article (including us). The reason is - copyright infringement in US and EU.

You can return only chunk of it

At least, that's what all lawyers I spoke to told me

Post reply on HN