Show HN: News Searching API

#1

Post by caballeto » Tue, Jul 07, 2020, 11:26 AM UTC

Re: Show HN: News Searching API

#2

Post by caballeto » Tue, Jul 07, 2020, 11:26 AM UTC

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-500$ per month, and I thought I could build and deploy such a system for a much cheaper price and improve my programming skills along the way. I would very much appreciate any comments/suggestions about the serive or how to make it better :)

Re: Show HN: News Searching API

#3

Post by sbr464 » Tue, Jul 07, 2020, 1:42 PM UTC

caballeto wrote: post #2

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-…

Nice work. Will share feedback soon after testing.

Re: Show HN: News Searching API

#4

Post by asdkhadsj » Tue, Jul 07, 2020, 1:58 PM UTC

Do you host the content as well? What's the legality of that?

I ask because I want to embed some archiving and "reader mode" logic into an app of mine that would be FOSS and self hosted. However that means each individual would be effectively scraping and archiving, and possibly p2p spreading, news content(as data sources).

So I'm curious if there is some underlying "fair use"-like mechanism that allows Archive, Outline.com, and you to consume news content without it being considered piracy.

Re: Show HN: News Searching API

#5

Post by bvm » Tue, Jul 07, 2020, 2:21 PM UTC

caballeto wrote: post #2

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-…

How are you licensing the content?

Re: Show HN: News Searching API

#6

Post by moneywoes » Tue, Jul 07, 2020, 3:57 PM UTC

Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?

Re: Show HN: News Searching API

#7

Post by caballeto » Tue, Jul 07, 2020, 4:35 PM UTC

sbr464 wrote: post #3

caballeto wrote: post #2

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-…

Nice work. Will share feedback soon after testing.

Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.

Re: Show HN: News Searching API

#8

Post by jslakro » Tue, Jul 07, 2020, 4:39 PM UTC

How do you get news from its sources?

Re: Show HN: News Searching API

#9

Post by caballeto » Tue, Jul 07, 2020, 4:39 PM UTC

moneywoes wrote: post #6

Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?

The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.

Re: Show HN: News Searching API

#10

Post by caballeto » Tue, Jul 07, 2020, 4:44 PM UTC

moneywoes wrote: post #6

Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?

No, I don't use RSS feeds. I extract news links from the home page and sitemaps mainly. Most of the news articles have a news schema embedded within them. You can check info about it at schema.org. This is a piece of JSON which contains the description of the article, usually title, publish date, description & image links.