Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: News Searching API

datanews.io

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Show HN: News Searching API

#21

Post by artembugara »

caballeto wrote: post #11
asdkhadsj wrote: post #4

Do you host the content as well? What's the legality of that? I ask because I want to embed some archiving and "reader mode" logic into an app of mine that would be FOSS and self hosted. However that means each individual would be effectively scraping and archiving, and possibly p2p spreading, news content (as data sources) . So I'm curious if there is some underlying "fair use"-like mechanism that allows Archive, Ou…

This is a great question. Thus far I thought, that the content that is publicly available without any limitation (e.g. membership access), can be scraped by anyone. You can take a look at hiQ Labs vs. LinkedIn [1]. LinkedIn's public data was scraped by data analytics company, the ruling was against LinkedIn. I also think that the use case matters, I don't republish their content on the site, but merely provide it via…

hiQ Labs vs. LinkedIn is a bad example. It is super specific.

LinkedIn did not owe the data (it was users').

In your case, you are reselling copyrighted product.

Of course you can scrape. It does not mean you can distribute this.

Re: Show HN: News Searching API

#22

Post by bvm »

caballeto wrote: post #15
jwhiz22 wrote: post #13

Earlier quoted context omitted.

He's not.

Right, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.

It's a pretty fine line you're treading here. A lot of the reason the competition is so expensive is that they are licensing the data in bulk. There's lots of litigation in this area, see the Meltwater cases in US and UK.

Re: Show HN: News Searching API

#23

Post by bvm »

artembugara wrote: post #20

Disclaimer: I am a co-founder of a similar News API service https://newscatcherapi.com/ No news API solution returns the full body text of the article (including us). The reason is - copyright infringement in US and EU. You can return only chunk of it At least, that's what all lawyers I spoke to told me

Some do, you just have to pay for it.

Re: Show HN: News Searching API

#24

Post by TechBro8615 »

caballeto wrote: post #9
moneywoes wrote: post #6

Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?

The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.

Nitpick: it’s “scrapes,” not “scraps.” To scrap something is to throw it away. To scrape something is to ingest it. The two words are, in fact, nearly opposite.

Sorry — I don’t mean to pick on you — it’s just that I see this mistake with an inexplicably high frequency, and it’s like nails on a chalkboard for me.

Also, very cool service!

Re: Show HN: News Searching API

#25

Post by fluffernutter »

caballeto wrote: post #16
ibdf wrote: post #14

Tried it.. pretty straight forward. Free account rate is good for personal use. I think it comes down to 4 calls per day which is reasonable if you are using this as maybe your daily news fetcher.

Thanks for the feedback! Concerning rate limiting, I think I may have placed rules a bit too strict, as the system is hosted on AWS Free Tier. There is definitely a room for increase in RPS per user, but I wanted to play safe.

This is great!

Re: Show HN: News Searching API

#26

Post by xiekomb »

Hi,

Simple and effective API. The documentation lacks details about the query parameter. I want to search for A AND B but q=?A%20B does not yield the expected result (seems a OR query) or for exact phrase "A invests in B" do not seem to work. Can you please post details for advanced syntax like this?

Re: Show HN: News Searching API

#27

Post by caballeto »

artembugara wrote: post #20

Disclaimer: I am a co-founder of a similar News API service https://newscatcherapi.com/ No news API solution returns the full body text of the article (including us). The reason is - copyright infringement in US and EU. You can return only chunk of it At least, that's what all lawyers I spoke to told me

That's interesting. I've browsed a few of alternative apis, and if I get it correctly then newsapi.org returns full text articles for paid plans. I would research this question further, and maybe will change the api to return only part of the content.

Re: Show HN: News Searching API

#28

Post by caballeto »

bvm wrote: post #22
caballeto wrote: post #15

Earlier quoted context omitted.

Right, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.

It's a pretty fine line you're treading here. A lot of the reason the competition is so expensive is that they are licensing the data in bulk. There's lots of litigation in this area, see the Meltwater cases in US and UK.

If you have say 10k news sources from tens of countries around the globe, I doubt that it would be feasible to contract with every specific site out there, especially if you get data say from Google News.

Re: Show HN: News Searching API

#29

Post by caballeto »

xiekomb wrote: post #26

Hi, Simple and effective API. The documentation lacks details about the query parameter. I want to search for A AND B but q=?A%20B does not yield the expected result (seems a OR query) or for exact phrase "A invests in B" do not seem to work. Can you please post details for advanced syntax like this?

Hi, thanks for the feedback! So far, I haven't implemented this feature yet. The query parameter gets tokenized as is, and then tokens are used to search in the index. Could you provide more details about your use case? As I understand you need to have AND, OR, NOT, grouping () operators, anything else? I would try to implement this today, and write back to you.

Re: Show HN: News Searching API

#30

Post by caballeto »

TechBro8615 wrote: post #24
caballeto wrote: post #9

Earlier quoted context omitted.

The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.

Nitpick: it’s “scrapes,” not “scraps.” To scrap something is to throw it away. To scrape something is to ingest it. The two words are, in fact, nearly opposite. Sorry — I don’t mean to pick on you — it’s just that I see this mistake with an inexplicably high frequency, and it’s like nails on a chalkboard for me. Also, very cool service!

Thanks, will take take a note for future.
Post reply on HN