Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Text RSS Feed: Get the whole feed and nothing but the feed

fulltextrssfeed.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#61

Post by nikcub »

yesimahuman wrote: post #39
grayrest wrote: post #11

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

Boilerpipe is by far the best tool for this that I've ever found ( http://code.google.com/p/boilerpipe/ ). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

Thanks for that link - exactly what I was looking for

btw I know that at Techmeme, Gabe spent years perfecting his story parsing for the 50k+ sites he tracks. Even something that would seem simple such as parsing the date of a story from a webpage has a ridiculous number of permutations that you have to grep for.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#63

Post by gregorym »

yesimahuman wrote: post #39
grayrest wrote: post #11

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

Boilerpipe is by far the best tool for this that I've ever found ( http://code.google.com/p/boilerpipe/ ). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

That is freaking awesome! http://boilerpipe-web.appspot.com/

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#64

Post by osmandros »

yesimahuman wrote: post #39
grayrest wrote: post #11

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

Boilerpipe is by far the best tool for this that I've ever found ( http://code.google.com/p/boilerpipe/ ). I'd be interested to hear if he is using something better, but I'd be surprised if he is. I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

That library is very robust. On the company I was working last year, We built a news crawler using that tool, and adapted it as a plugin for nutch.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#65

Post by lurchpop »

nicpottier wrote: post #60

We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done. Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to…

how do you handle scale like that? Do you have a hadoop cluster or something? How many concurrent do you download?

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#66

Post by hokkos »

Is it legal ? Can you legally copy all the content of a site and publish it while striping the ads ?

I've tough of this idea since 2 years, but I am so ineffective at building my own ideas that it doesn't surprise me that someone else built it, as the idea was really floating more and more since instapaper mobilizer.

Considering the legal aspect I had more ideas about that. It is to hide behind the DMCA takedown, and provide an email address to take-down a feed. But do not map the www.example.com/feed.xml to http://fulltextrssfeed.com/www.example.com/feed.xml , but use an alias, so the take-down just remove the alias not the whole * .example.com*.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#67

Post by nicpottier »

lurchpop wrote: post #65
nicpottier wrote: post #60

We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done. Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to…

how do you handle scale like that? Do you have a hadoop cluster or something? How many concurrent do you download?

Amazon EC2, MongoDB, S3. The EC2 instances scale with how many stale feeds we have, but it is usually less than 2.

Just checked and we have ~25k feeds in the system, though not all are deep harvesting as we call it.

Note we do a few things over just extracting the full content as well, we also try to grab out images and create a pleasing thumbnail using face detection etc.. So that probably slows things down a good deal as well.

Re: Full Text RSS Feed: Get the whole feed and nothing but the feed

#68

Post by tianyicui »

nicpottier wrote: post #60

We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done. Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to…

I really wish one day Google will sell a page harvesting service. It can certainly profit since the cost is neglectable. But how big the market is?
Post reply on HN