Show HN: News Searching API

Post by xiekomb » Wed, Jul 08, 2020, 11:40 AM UTC

caballeto wrote: post #29

xiekomb wrote: post #26

Hi, Simple and effective API. The documentation lacks details about the query parameter. I want to search for A AND B but q=?A%20B does not yield the expected result (seems a OR query) or for exact phrase "A invests in B" do not seem to work. Can you please post details for advanced syntax like this?

Hi, thanks for the feedback! So far, I haven't implemented this feature yet. The query parameter gets tokenized as is, and then tokens are used to search in the index. Could you provide more details about your use case? As I understand you need to have AND, OR, NOT, grouping () operators, anything else? I would try to implement this today, and write back to you.

Post by xiekomb » Wed, Jul 08, 2020, 1:41 PM UTC

caballeto wrote: post #32

xiekomb wrote: post #31

Earlier quoted context omitted.

yes this is exactly what I (most users?) would need. Search for exact phrase would be a plus also. Eg: "donald trump" instead of Donald AND trump. What kind of backend are you using, a simple database or search index like elasticsearch (which would be more appropriate for such project)?

Hi, just finished implementing this stuff. To search a AND b use 'q=A+B'. Here are a few examples: > curl -XGET -G 'api.datanews.io/v1/news' -d 'apiKey=API_KEY' --data-urlencode 'q="Europe throws new rule book at Google, tech giants to loosen market grip"' > curl -XGET -G 'api.datanews.io/v1/news' -d 'apiKey=API_KEY' --data-urlencode 'q=google+facebook+amazon' Supported operators: + AND, | OR, - NOT, () parentheses f…

Post by graupel » Thu, Jul 09, 2020, 8:13 PM UTC

Post by caballeto » Thu, Jul 09, 2020, 9:38 PM UTC

erwinkle wrote: post #34

caballeto wrote: post #12

Earlier quoted context omitted.

First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.

Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?

Post by hboon » Fri, Jul 10, 2020, 5:52 AM UTC

artembugara wrote: post #21

caballeto wrote: post #11

Earlier quoted context omitted.

This is a great question. Thus far I thought, that the content that is publicly available without any limitation (e.g. membership access), can be scraped by anyone. You can take a look at hiQ Labs vs. LinkedIn [1]. LinkedIn's public data was scraped by data analytics company, the ruling was against LinkedIn. I also think that the use case matters, I don't republish their content on the site, but merely provide it via…

hiQ Labs vs. LinkedIn is a bad example. It is super specific. LinkedIn did not owe the data (it was users'). In your case, you are reselling copyrighted product. Of course you can scrape. It does not mean you can distribute this.