Live data from Hacker News

Show HN: Python package to collect normalized news from almost any website

github.com

191–200 of 203 posts

Re: Show HN: Python package to collect normalized news from almost any website

#191

This looks like a cool library, but I have a question about the newscatcher API. How does the licensing work for the content? Seems odd (but great) that I can just read the news in my terminal from NYT but not pay a subscription or see ads. I read in some of the comments it's an RSS feed, is that freely available all the time? Surely even the RSS feed is protected with copyright and has restrictions on republishing?…

Hi. Co-founder is here. Short answer. We do not know if it is legal!

It seems a little risky to build a paid service that you're not sure is legal.

Also, the Terms of Service, Privacy Policy, and GDPR Policy links in the footer of your site don't work. They all have empty hrefs.

Re: Show HN: Python package to collect normalized news from almost any website

#192
post #157

Earlier quoted context omitted.

When I was a kid CNN used to do this: they would invite a single climate scientist and a single climate change skeptic and have them make their points on equal time. CNN would then say “who’s right? You decide.” And the program would end. This was extremely harmful to the overall mission of informing people and created a false balance between sides which are not equally valid. Today, this is almost universally seen a…

I am showing my age when I remember PBS having an entire show where two sides discussed an important issue of domestic or international concern. It was called "The Advocates"[0]. It ran for five years, and included a moderator and two teams that basically had a debate as a TV show. Michael Dukakis, who later was the Democratic nominee for president, was one of the moderators. (Dukakis' Wikipedia page has no mention t…

I used to watch some of these IQ debates in college, they can be really good.

The longer format and strong moderation really lend themselves to interesting discussions, and you can learn a lot. One of my favourite debates of this format (though not IQ) is Peter Thiel vs Eric Schmidt (then executive chairman of Google) arguing about whether Google was still capable of innovation [2012]:

https://youtu.be/PsXFwy6gG_4

Re: Show HN: Python package to collect normalized news from almost any website

#193

Earlier quoted context omitted.

Do you guys have any plans to create webhooks that let you know when a feed/search is updated with a new article in real time?

Hey. Not yet. Let us think about that.

It's probably not worth it. Anyone needing near-real-time feed webhook updates will probably build their own scraper.

Re: Show HN: Python package to collect normalized news from almost any website

#194

Earlier quoted context omitted.

When I was a kid CNN used to do this: they would invite a single climate scientist and a single climate change skeptic and have them make their points on equal time. CNN would then say “who’s right? You decide.” And the program would end. This was extremely harmful to the overall mission of informing people and created a false balance between sides which are not equally valid. Today, this is almost universally seen a…

> This was extremely harmful to the overall mission of informing people and created a false balance between sides which are not equally valid. Today, this is almost universally seen as a failure of journalism. So CNN should get to decide what is correct? Go fuck yourself, and learn how to think. If CNN's audiance can't make their own decisions then that is the fault of the audience not CNN. We don't need, and should…

> Go fuck yourself, and learn how to think

Ouch. Accounts that post like this get banned here regardless of how wrong someone else is or you feel they are. Please review https://news.ycombinator.com/newsguidelines.html and please don't ever do this here again. (We had to warn you about personal attacks once before.)

Re: Show HN: Python package to collect normalized news from almost any website

#195
post #44

This seems really cool! How does it work internally? Is it downloading the news from a RSS or is it crawling the content of the website? Or is the content coming from an external service? How are the feeds selected? Can we add more? who is maintaining them (in case the data is crawled)? Thanks!

hey, there is a sqlite DB that stores the RSS endpoints. Then we use feedparser python package to parse it.

I see, thanks for the answer and thanks for sharing!

Re: Show HN: Python package to collect normalized news from almost any website

#196

Earlier quoted context omitted.

Hi. Co-founder is here. Short answer. We do not know if it is legal!

It seems a little risky to build a paid service that you're not sure is legal. Also, the Terms of Service, Privacy Policy, and GDPR Policy links in the footer of your site don't work. They all have empty hrefs.

Will be updated once we start to sell

Re: Show HN: Python package to collect normalized news from almost any website

#197
As others have noted, this doesn't seem to collect the full article text, just stuff that you would get from an RSS fee. From the title, I expected something more like newspaper3k[0]. I used that for an NLP class during undergrad to collect full-text news articles, in conjunction with Selenium (many mainstream sites don't work with just plain wget or requests).

Lately I've starting using EpubPress[1] to grab full-text articles and generate an ePub, which happens every night via cron. Then I can get a full digest on my iPad over sftp at my leisure. Sadly EpubPress is not very sophisticated, sites like Bloomberg or ArsTechnica return "are you a robot" challenges which it can't bypass.

I wish there was some kind of community driven library for retrieving full-text articles from common sites. In my vision of how that would work, users would contribute hand-crafted Selenium scripts to download and extract the article text, bypassing the bot-detection for each site. Then something like EpubPress would work a lot better.

The "modern web" just has too much junk to be interesting any more. Sometimes news sites publish articles I would like to read, but I'm not interested in dealing with 1000 different implementations of crappy mobile UIs, advertisements, animations, etc. I know reader view exists, but you still have to wait for the page to load, and it doesn't work very well for some sites. For the sites it doesn't break with, the experience with EpubPress is much better.

0 - https://github.com/codelucas/newspaper

1 - https://epub.press/

Re: Show HN: Python package to collect normalized news from almost any website

#198

Earlier quoted context omitted.

I’m sure this is not an original comment, but it’s interesting to see what you’ve classified as left/right. It must be difficult given that there is not really one axis of left/right and that “the centre” is highly relative. To me, seeing the BBC and Euronews in the “left” section is pretty funny, but I guess it’s true relative to US politics. Is there anything you’ve learned about “the left media” and “the right med…

The fact US politics call them "left" and "right" is meaningless even, given how right-shifted US politics are (By EU standards, only US extreme-progressives are actually in the european "Left"). Not to mention americans have demonized "the center" as some "if you're somehow trying to consider all the facts you're a coward who can't decide" type of thing. The two sides being "at war" drives TV/website engagement and…

[dead]

Re: Show HN: Python package to collect normalized news from almost any website

#199
post #40

Earlier quoted context omitted.

Educating yourself about every topic out there is absolutely impossible. Thats precisely why news businesses exist in the first place.

> Thats precisely why news businesses exist in the first place. Nah. My view is that news exist for entertainment. That might not be their stated goal, but it's why people consume them.

I do not claim they educate but benefit from the mainstream ignorance about a vast variety of topics.
Post reply on HN