Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: RSS feeds for arbitrary websites using CSS selectors

feed-me-up-scotty.vincenttunru.com

Post reply on HN
Print view View original

11–20 of 134 posts

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#11

Post by axiolite »

Vinnl wrote: post #8
k1m wrote: post #7

Very nice! I work on Feed Creator - https://createfeed.fivefilters.org - which is similar, although unlike yours doesn't use a headless browser, so selecting Javacript-inserted elements isn't supported.

That looks great! I ran into a couple of feed generators when I first needed this myself, but they were either paid or clumsy to use. That's when I thought, "alright, I can build this myself, and then I can even use CSS selectors, since I'm comfortable with those anyway". Of course, that is the point I should've thought about explicitly searching for one that supported that :) Ah well. What I do like about my current…

Here's a list of RSS feed generators:

https://github.com/AboutRSS/ALL-about-RSS#webpagehtml

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#12

Post by Vinnl »

moehm wrote: post #9
Vinnl wrote: post #5

Earlier quoted context omitted.

That's a good one, I'll see if I can add something to the site. Meanwhile you can see the generated examples here: - https://vincenttunru.gitlab.io/feeds/funfacts.xml - https://vincenttunru.gitlab.io/feeds/wikivoyage.xml And the combined feed: - https://vincenttunru.gitlab.io/feeds/all.xml Add those links to your feed reader to see rendered examples while I update the site. Edit: preview added to the website.

FYI, the links inside your feeds don't work, because they are relative, not absolute.

Ah, you mean the ones inside the contents? That's a good one. I'm not sure if that's easily fixable, but I'll give it some thought. For those interested, I'll track it here: https://gitlab.com/vincenttunru/feed-me-up-scotty/-/issues/1

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#13

Post by axiolite »

Vinnl wrote: post #6
rcarmo wrote: post #3

This is nice. I’m actually doing much the same with Node-RED’s HTML parser, which also supports simple selectors.

Thanks. I initially used a regular HTML parser as well, but I quickly ran into sites that wouldn't render without JavaScript. I'm therefore now using a regular browser controlled by Playwright to fetch the websites.

Care to name any sites? I've always managed to find workarounds for everything I've wanted to follow. Most websites want to be indexed by search engines, and googlebot doesn't do javascript. So sometimes a forged user agent is all you need. Occasionally, finding the actual json file and parsing the info you need out of it does the job. etc.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#14

Post by chrismorgan »

Vinnl wrote: post #12
moehm wrote: post #9

Earlier quoted context omitted.

FYI, the links inside your feeds don't work, because they are relative, not absolute.

Ah, you mean the ones inside the contents? That's a good one. I'm not sure if that's easily fixable, but I'll give it some thought. For those interested, I'll track it here: https://gitlab.com/vincenttunru/feed-me-up-scotty/-/issues/1

The appropriate way is to use the xml:base attribute, as demonstrated in the example at https://datatracker.ietf.org/doc/html/rfc4287#page-4.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#15

Post by k1m »

Vinnl wrote: post #8
k1m wrote: post #7

Very nice! I work on Feed Creator - https://createfeed.fivefilters.org - which is similar, although unlike yours doesn't use a headless browser, so selecting Javacript-inserted elements isn't supported.

That looks great! I ran into a couple of feed generators when I first needed this myself, but they were either paid or clumsy to use. That's when I thought, "alright, I can build this myself, and then I can even use CSS selectors, since I'm comfortable with those anyway". Of course, that is the point I should've thought about explicitly searching for one that supported that :) Ah well. What I do like about my current…

The more the merrier! :) And always nice to see more in the RSS space.

Feed Creator is in PHP and unlike yours doesn't produce a static file. The CSS selectors, URL and other parameters are all embedded in the URL, e.g.

  https://createfeed.fivefilters.org/extract.php?url=http%3A%2F%2Fjohnpilger.com%2Farticles&item=.entry&item_desc=.intro&item_date=.entry-date
The biggest issue I've found is people struggling to work out which CSS selectors to use. I wrote a blog post not that long ago to help people use the browser's developer tools to do that. Might help too with anyone trying to use this (although perhaps most of the HN audience doesn't need help here): https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-...

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#17

Post by moehm »

axiolite wrote: post #13
Vinnl wrote: post #6

Earlier quoted context omitted.

Thanks. I initially used a regular HTML parser as well, but I quickly ran into sites that wouldn't render without JavaScript. I'm therefore now using a regular browser controlled by Playwright to fetch the websites.

Care to name any sites? I've always managed to find workarounds for everything I've wanted to follow. Most websites want to be indexed by search engines, and googlebot doesn't do javascript. So sometimes a forged user agent is all you need. Occasionally, finding the actual json file and parsing the info you need out of it does the job. etc.

> googlebot doesn't do javascript

But it does.

Primary source: I experienced it on my own site.

Secondary source: https://developers.google.com/search/docs/guides/fix-search-...

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#18

Post by chrismorgan »

I’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#19

Post by nvr219 »

chrismorgan wrote: post #18

I’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.

If OP didn't use the word RSS and said "atom feeds" I wouldn't know what they're talking about, so I recommend keep calling it RSS.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#20

Post by chrismorgan »

moehm wrote: post #17
axiolite wrote: post #13

Earlier quoted context omitted.

Care to name any sites? I've always managed to find workarounds for everything I've wanted to follow. Most websites want to be indexed by search engines, and googlebot doesn't do javascript. So sometimes a forged user agent is all you need. Occasionally, finding the actual json file and parsing the info you need out of it does the job. etc.

> googlebot doesn't do javascript But it does. Primary source: I experienced it on my own site. Secondary source: https://developers.google.com/search/docs/guides/fix-search-...

This changes from time to time, of course, but when last I investigated, around two years ago, consensus was that it mostly wouldn’t do JavaScript until you nudged it into doing so in some way that I forget, and that it was always slower to index/update if it needed to do JavaScript.

(For my part, I disable JavaScript by default for various reasons, mostly performance, and it’s decidedly uncommon for a general-internet site to be completely broken by it. Sites that get posted on HN are disproportionately JS-dependent, especially if they’re new.)

Post reply on HN