Show HN: RSS feeds for arbitrary websites using CSS selectors • Hacker News Board

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#71

Post by spacec0wb0y » Mon, Jul 05, 2021, 8:14 PM UTC

Vinnl wrote: post #68

spacec0wb0y wrote: post #60

What do I do to get this working? So far I've forked feeds, edited feeds.toml, checked it out as a branch gh-pages, pushed the branch up to github. I can see the page at .github.io/feeds/" rel="nofollow">https:// .github.io/feeds/ but .github.io/feeds/actions" rel="nofollow">https:// .github.io/feeds/actions is just a 404.

Ah, those instructions are unclear — as far as I know, you first have to go to /feeds/actions" rel="nofollow">https://github.com/ /feeds/actions to enable Workflows for your repository. Then, your feeds should be published to .github.io/feeds/ .xml" rel="nofollow">https:// .github.io/feeds/ .xml . Does that work?

not exactly. I saw below the instruction to run `npx feed-me-up-scotty` so I did and it generated the public/ dir and feeds.

ok I managed to access the actions route by a different URL to the readme. I copied the pages.yml workflow from your repo.

Few minutes later I could see my feed. Very nice! I need to clean up my selectors now!

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#72

Post by ricardo81 » Mon, Jul 05, 2021, 8:38 PM UTC

detaro wrote: post #61

ricardo81 wrote: post #58

Earlier quoted context omitted.

Yes, if you're determined enough you can scrape anything. Just not sure what the new thing here is.

who has claimed there is something fundamentally new here?

Presenting something on the home page of HN would suggest something novel. I must be missing the point because of the downvotes. Happy to be enlightened. Pick CSS selectors and scrape something? That's quintessentially scraping.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#73

Post by remram » Mon, Jul 05, 2021, 8:39 PM UTC

Couldn't those selectors be maintained by the community? Instead of everyone deploying this on their own GitHub Actions, and having to fix it independently when it breaks, a single repo with all kinds of feeds maintained by everyone?

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#74

Post by cyborgx7 » Mon, Jul 05, 2021, 8:53 PM UTC

this is something I've been looking for

neat

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#75

Post by Vinnl » Mon, Jul 05, 2021, 8:55 PM UTC

remram wrote: post #73

Couldn't those selectors be maintained by the community? Instead of everyone deploying this on their own GitHub Actions, and having to fix it independently when it breaks, a single repo with all kinds of feeds maintained by everyone?

That's an interesting idea: something like DefinitelyTyped, but instead of type definitions for npm packages it provides selectors for URLs. Main challenge there would be organising the moderation, I suppose.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#76

Post by fran-penedo » Mon, Jul 05, 2021, 9:03 PM UTC

k1m wrote: post #7

Very nice! I work on Feed Creator - https://createfeed.fivefilters.org - which is similar, although unlike yours doesn't use a headless browser, so selecting Javacript-inserted elements isn't supported.

Since everyone is pitching their own, I built https://github.com/fran-penedo/rssify, which started as a fork of https://github.com/h43z/rssify. The basic functionality is similar to Vinnl's: give it a URL and some selectors and it builds the RSS feed. From this, I added a few things: templates (if you want to subscribe to individual projects within a webpage, like fanfics in ao3), transforms (when the data is not quite the text of the DOM element), a flask server you can use to add new URLs you have a template for and update the feeds, and a userscript to add the current URL using the server.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#77

Post by z3c0 » Mon, Jul 05, 2021, 9:10 PM UTC

cromka wrote: post #69

ricardo81 wrote: post #41

Earlier quoted context omitted.

Pick a number of programming languages and you can scrape a page in 10-20 lines of code in many cases. The barrier to entry is understanding the DOM layout for a particular website, which is subject to change at any moment. Purely IMO, a more friendly way to go about it to abstract from code and CSS knowledge is to run a UI that highlights elements, lets you select them, select the title, description, link etc and th…

> highlights elements, lets you select them, And how would that software then remember your chosen elements if not by their CSS ids/classes?

By positional indexing, of course.

Just kidding, that's horrendous.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#78

Post by ricardo81 » Mon, Jul 05, 2021, 9:13 PM UTC

cromka wrote: post #69

ricardo81 wrote: post #41

Earlier quoted context omitted.

Pick a number of programming languages and you can scrape a page in 10-20 lines of code in many cases. The barrier to entry is understanding the DOM layout for a particular website, which is subject to change at any moment. Purely IMO, a more friendly way to go about it to abstract from code and CSS knowledge is to run a UI that highlights elements, lets you select them, select the title, description, link etc and th…

> highlights elements, lets you select them, And how would that software then remember your chosen elements if not by their CSS ids/classes?

You must have missed the point, or I'm missing the point of this thing. Select whatever you like, do I need an external tool to do that? Either I have knowledge of the DOM or I don't, and if I don't then a UI selector would be the next best guess.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#79

Post by contingencies » Mon, Jul 05, 2021, 9:18 PM UTC

Vinnl wrote: post #75

remram wrote: post #73

Couldn't those selectors be maintained by the community? Instead of everyone deploying this on their own GitHub Actions, and having to fix it independently when it breaks, a single repo with all kinds of feeds maintained by everyone?

That's an interesting idea: something like DefinitelyTyped, but instead of type definitions for npm packages it provides selectors for URLs. Main challenge there would be organising the moderation, I suppose.

It should be feasible to analyse the structure over time of the extracted data. Therefore, any proposed change which breaks the anticipated rhythm would be suspect.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#80

Post by KirillPanov » Mon, Jul 05, 2021, 9:22 PM UTC

lhnz wrote: post #50

ricardo81 wrote: post #30

Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.

If you scrape using accessible parts of the page (e.g. `aria-` attributes) it is less likely to be brittle, since if it were it would mean their site had stopped being accessible to screen readers, etc.

Why the downvotes here? This is objectively true and, in my experience, very very useful.

Screen readers are scrapers too. "Scraper" does not mean "user agent I don't like".