Full Text RSS Feed: Get the whole feed and nothing but the feed • Hacker News Board

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#51

Post by beagledude » Wed, Mar 02, 2011, 3:52 AM UTC

grayrest wrote: post #11

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

Goose article extractor has a full suite of unit tests and also does pure text and image extractions: https://github.com/jiminoc/goose

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#52

Post by yahelc » Wed, Mar 02, 2011, 3:58 AM UTC

Considering the impending lawyer-takedown, it would be great if this was made open source, so people can implement their own local versions on their own servers.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#53

Post by yahelc » Wed, Mar 02, 2011, 4:00 AM UTC

AdamGibbins wrote: post #4

Excellent thanks, shall be applying this to all my Gawker feeds.

You know they make a full-feed version available of all their sites, right? It's just of the form http://lifehacker.com/vip.xml

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#54

Post by yahelc » Wed, Mar 02, 2011, 4:00 AM UTC

matsur wrote: post #50

guptaneil wrote: post #33

Earlier quoted context omitted.

The preview for Lifehacker returned nothing at all, but adding the feed to Google Reader worked as advertised. I guess, don't rely on the preview box.

This may be common knowledge, but all gawker blogs are available in full feed, ad free form at .com/vip.xml i.e. http://lifehacker.com/vip.xml

Ah, I see I'm not the first to point this out :)

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#55

Post by benmccann » Wed, Mar 02, 2011, 4:09 AM UTC

yesimahuman wrote: post #39

grayrest wrote: post #11

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

Boilerpipe is by far the best tool for this that I've ever found ( http://code.google.com/p/boilerpipe/ ). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#56

Post by cvandyck76 » Wed, Mar 02, 2011, 4:41 AM UTC

Is there an argument to be made that the content providers only get 'paid' if the RSS reader is enticed to click through to the site? I'm all for neat services, but I think that this is a little bit unfair to the other party.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#57

Post by sankara » Wed, Mar 02, 2011, 4:49 AM UTC

Works with BBC. Thanks.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#58

Post by yesimahuman » Wed, Mar 02, 2011, 5:47 AM UTC

benmccann wrote: post #55

yesimahuman wrote: post #39

Earlier quoted context omitted.

Boilerpipe is by far the best tool for this that I've ever found ( http://code.google.com/p/boilerpipe/ ). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.

Yea, I'd be curious to see exactly what he's doing. I can only guess there is a heuristic which results in a lot of failed feed processing noticed on here (I know it's just a weekend project :)) that doesn't generalize well. Boilerpipe, in my experience, works very well on almost all news/blog type content. Finding the date in the first few sentences and the title are extra heuristics that can be added later.

EDIT: The date and title are in the RSS feed already! No further analysis needed.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#59

Post by iphoneedbot » Wed, Mar 02, 2011, 5:52 AM UTC

I love it! It works for tumblr rss -- I really wish though that it you can opensource it. (Well, I would just hate it if you start having hosting problems or other problems that would cause you the need to shut down)

Im currently using "Readable Feeds" Nirmal J. Patel (http://www.nirmalpatel.com/hacks/hnrss.html) and Andrew Trusty (http://andrewtrusty.com/2009/06/29/readable-feeds/)

I like it -- but its really inconsistent!

Cheers,

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#60

Post by nicpottier » Wed, Mar 02, 2011, 6:24 AM UTC

We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done.

Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to keep them all up to date. Storing and serving it is no big deal, but harvesting actually turns into real money when you figure out total bandwidth used. (we harvest about ~30,000 feeds every 15 minutes)