Live data from Hacker News

RSS Box – RSS for websites that do not support RSS

rssbox.herokuapp.com

11–20 of 110 posts

Re: RSS Box – RSS for websites that do not support RSS

#11
post #7

this is really cool. Twitter didn't work, though.

For twitter I use the perl backend scripts (https://github.com/ciderpunx/twitrssme/tree/master/fcgi) from http://twitrss.me/ by itself. It's pretty easy to scrape twitter users/searches and generate RSS feeds on disk for my native reader.

# in a bash script called by cron every handful of hours there are many, many lines like this:

    perl twitter_user_to_rss.pl gnuradio > ~/limbo/www/rss/gnuradio.xml

    perl twitter_search_to_rss_wtf.pl "rtlsdr" > ~/limbo/www/rss/rtlsdr.xml

Re: RSS Box – RSS for websites that do not support RSS

#12
post #2

> FYI: If you enable JavaScript then you be able to access additional options in the dropdown menus. The website should still be somewhat usable, but recent versions of Firefox will try to download the RSS feeds. This is how you communicate with people with JavaScript disabled! Kudos. Most sites either present you with a blank page with no information, a blank page asking to enable JavaScript (even when the content i…

> What if we don’t have control over the frequency of requests (e.g. using a service like Feedly)?

I believe most RSS reader-clients and aggregator backends are programmed to respect HTTP cache-control headers; so as long as the developer of this service sets those headers appropriately for their endpoints, there shouldn't be a problem.

The warning is likely more for people's custom scripting using curl(1) et al, where there isn't an HTTP cache in the code path.

Re: RSS Box – RSS for websites that do not support RSS

#15
Anyone ever did something like this for Facebook?

I know it'd have to be subjective per user (security ACLs ⇒ different accounts seeing differing subsets of other accounts' posts); but I'd be fine with just getting my own account's subjective view, by logging into such a service using Facebook OAuth (or, if that isn't enough, then I'd be fine with handing over my Facebook creds themselves, ala XAuth, provided the service is a FOSS one I'm running a copy of myself in e.g. an ownCloud instance.)

I also know that it'd likely require heavyweight scraping using e.g. Puppeteer, to fool Facebook into thinking it's real traffic. But that's not really that much of an impediment, as long as you don't need to scale it to more than a dozen-or-so scrapes per second. (Which you'd automatically be safe from if it was a host-it-yourself solution, since there'd only be one concurrent user of your instance.)

Anyone done this?

Re: RSS Box – RSS for websites that do not support RSS

#19
If you want to process an RSS feed programmatically, you have to run code to poll the feed and keep track of items already processed. This isn't hard to write, but it's often not core to your app's logic.

You probably just want to run code on each new item in the feed.

Pipedream lets you treat an RSS feed as an event source. Pipedream runs the code to poll the feed, emitting new items as the feed produces them.

RSS for Hackers - https://rss.pipedream.com

Re: RSS Box – RSS for websites that do not support RSS

#20
post #2

> FYI: If you enable JavaScript then you be able to access additional options in the dropdown menus. The website should still be somewhat usable, but recent versions of Firefox will try to download the RSS feeds. This is how you communicate with people with JavaScript disabled! Kudos. Most sites either present you with a blank page with no information, a blank page asking to enable JavaScript (even when the content i…

Feed aggregation services tend to minimize the frequency of requests especially for unpopular feeds since more requests = higher load for them as well. The incentives on the publishing and consuming sides align. Many services only offer to increase crawling frequencies for premium users, and even then only for a limited number of feeds. Not to mention they only need to crawl once for however many subscribers.

It’s really people who don’t use aggregation services and set their clients to update very frequently (say every minute) that pose a problem.

Post reply on HN