Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Python package to collect normalized news from almost any website

github.com

Post reply on HN
Print view View original

21–30 of 203 posts

Re: Show HN: Python package to collect normalized news from almost any website

#21

Post by pmyteh »

Depending on what you're trying to do, there's also newspaper3k: https://github.com/codelucas/newspaper

It's quite easy to get "good" extraction for large numbers of outlets/articles without a massive amount of special-casing, as news articles are nearly universally marked up with RDF metadata (partly for Google News's benefit). Article discovery, and perfect parsing, is quite a bit harder. I ended up rolling a new Scrapy project with site-specific parsing code for an academic project as I had quite specific requirements.

Re: Show HN: Python package to collect normalized news from almost any website

#22

Post by trabant00 »

doublesCs wrote: post #18
oceanbreeze83 wrote: post #5

For any news junkies heres, I've built https://maagnit.com which gathers both Left and Right leaning sources for any story and displays them altogether on one page. My approach has been, if we can't get neutral/objective coverage, getting comprehensive, 360 desgree coverage is a good alternative. These days, news bias is not only in the way a story is covered, but also which stories are covered. So, maagnit automatic…

Seems like a terrible idea to me. The two sides are at war, and if the way you sample them is by going 50-50, that encourages them to be more extreme because that will skew the average in their direction. We already see this in the media. You have the host and two people. Lets hear what person A says, lets hear what person B says. Who's right? Impossible to tell, otherwise you must be biased. And no mention that pers…

And the better alternative is...? What you say is correct but also worthless unless you provide a better way to do it.

Re: Show HN: Python package to collect normalized news from almost any website

#23

Post by p2detar »

trabant00 wrote: post #22
doublesCs wrote: post #18

Earlier quoted context omitted.

Seems like a terrible idea to me. The two sides are at war, and if the way you sample them is by going 50-50, that encourages them to be more extreme because that will skew the average in their direction. We already see this in the media. You have the host and two people. Lets hear what person A says, lets hear what person B says. Who's right? Impossible to tell, otherwise you must be biased. And no mention that pers…

And the better alternative is...? What you say is correct but also worthless unless you provide a better way to do it.

Definitely not worthless. Also OP makes a good point at the end:

> The thing is that to be able to tell the different takes effort in educating yourself.

Re: Show HN: Python package to collect normalized news from almost any website

#24

Post by rawoke083600 »

marban wrote: post #8

Self-Plug: https://www.hvper.com (Official Successor of popurls which more or less started the single page aggregator craze.)

Looks good ! How do I get the BIG pink bar away ? I would rather see ads (to support the free version) than that BIG PINK BAR ?

Re: Show HN: Python package to collect normalized news from almost any website

#25

Post by scrollaway »

heisenzombie wrote: post #11
oceanbreeze83 wrote: post #5

For any news junkies heres, I've built https://maagnit.com which gathers both Left and Right leaning sources for any story and displays them altogether on one page. My approach has been, if we can't get neutral/objective coverage, getting comprehensive, 360 desgree coverage is a good alternative. These days, news bias is not only in the way a story is covered, but also which stories are covered. So, maagnit automatic…

I’m sure this is not an original comment, but it’s interesting to see what you’ve classified as left/right. It must be difficult given that there is not really one axis of left/right and that “the centre” is highly relative. To me, seeing the BBC and Euronews in the “left” section is pretty funny, but I guess it’s true relative to US politics. Is there anything you’ve learned about “the left media” and “the right med…

The fact US politics call them "left" and "right" is meaningless even, given how right-shifted US politics are (By EU standards, only US extreme-progressives are actually in the european "Left").

Not to mention americans have demonized "the center" as some "if you're somehow trying to consider all the facts you're a coward who can't decide" type of thing. The two sides being "at war" drives TV/website engagement and that's all that matters to the people writing the headlines.

It just so happens that currently, one of the two "sides" relies heavily on disinformation; so anything that tries to fight disinformation (including remaining impartial) is that side's enemy. So those things end up being considered "left-wing".

Truly, the united states has four political parties: The Media Left, the Media Right, the Political Left, and the Political Right. Nearly every american you know is part of the first two; the last two don't make for good TV.

Re: Show HN: Python package to collect normalized news from almost any website

#26

Post by sabertoothed »

oceanbreeze83 wrote: post #5

For any news junkies heres, I've built https://maagnit.com which gathers both Left and Right leaning sources for any story and displays them altogether on one page. My approach has been, if we can't get neutral/objective coverage, getting comprehensive, 360 desgree coverage is a good alternative. These days, news bias is not only in the way a story is covered, but also which stories are covered. So, maagnit automatic…

The US really needs to change their political system so that it's no longer "the winner takes it all" politics where only two parties can effectively rival. This is so poisonous for everything. No nuances in the discussion. Truth comes under the wheels. Stupid slogans win. No chance for larger green/ecological movements besides the overwhelming topic of economics. Ridiculous gerrymandering. It's just very sad to see.

Re: Show HN: Python package to collect normalized news from almost any website

#27

Post by freddie_mercury »

sabertoothed wrote: post #26
oceanbreeze83 wrote: post #5

For any news junkies heres, I've built https://maagnit.com which gathers both Left and Right leaning sources for any story and displays them altogether on one page. My approach has been, if we can't get neutral/objective coverage, getting comprehensive, 360 desgree coverage is a good alternative. These days, news bias is not only in the way a story is covered, but also which stories are covered. So, maagnit automatic…

The US really needs to change their political system so that it's no longer "the winner takes it all" politics where only two parties can effectively rival. This is so poisonous for everything. No nuances in the discussion. Truth comes under the wheels. Stupid slogans win. No chance for larger green/ecological movements besides the overwhelming topic of economics. Ridiculous gerrymandering. It's just very sad to see.

So something like the super-dysfunctional political systems of Israel or Belgium or Italy or South Africa?

Fixing first past the post voting is not some magic cure-all.

Re: Show HN: Python package to collect normalized news from almost any website

#30

Post by k1m »

nmstoker wrote: post #19

How complete are the articles that are returned? Last time I looked into RSS news feeds a lot of sources would just put abbreviated / teaser content in and then try to get you to click through to the story on their site. That obviously didn't make for the best experience with an RSS downloader though.

I work on Full-Text RSS which can help convert abbreviated feeds into full-text versions. The idea is you'll get a new feed URL from Full-Text RSS to use instead of the original partial feed in your news reader or application.

Free to try here: http://ftr.fivefilters.org/ and code for a slightly older version available here: https://bitbucket.org/fivefilters/full-text-rss/src/master/

Post reply on HN