Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: RSS feeds for arbitrary websites using CSS selectors

feed-me-up-scotty.vincenttunru.com

Post reply on HN
Print view View original

21–30 of 134 posts

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#21

Post by moehm »

chrismorgan wrote: post #18

I’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.

Same problem with SSL encryption. But with both RSS and SSL, if the user use a modern client, he shouldn't bother if its internally using RSS, Atom, SSL or TLS.

I for one know most of my pals do know about RSS, but haven't heard the term 'Atom feed'.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#22

Post by Vinnl »

axiolite wrote: post #13
Vinnl wrote: post #6

Earlier quoted context omitted.

Thanks. I initially used a regular HTML parser as well, but I quickly ran into sites that wouldn't render without JavaScript. I'm therefore now using a regular browser controlled by Playwright to fetch the websites.

Care to name any sites? I've always managed to find workarounds for everything I've wanted to follow. Most websites want to be indexed by search engines, and googlebot doesn't do javascript. So sometimes a forged user agent is all you need. Occasionally, finding the actual json file and parsing the info you need out of it does the job. etc.

The User Agent trick is a good one that I should've tried, but I just checked and it didn't work for this one. Parsing actual JSON wasn't really an option, as I wanted to be able to quickly and easily add RSS feeds.

Possibly SEO is less a concern for the type of website I initially made this for, i.e. Dutch real estate agents. Most people find their listings through funda.nl rather than through search engines; I was just hoping to see them listed before they got posted there.

Send me a message on Twitter or email me (hacker_news@ my domain) if you still want the URL of a failing website to play around with.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#23

Post by chrismorgan »

moehm wrote: post #21
chrismorgan wrote: post #18

I’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.

Same problem with SSL encryption. But with both RSS and SSL, if the user use a modern client, he shouldn't bother if its internally using RSS, Atom, SSL or TLS. I for one know most of my pals do know about RSS, but haven't heard the term 'Atom feed'.

“SSL” is finally dying as a term; the significant majority of what I see calls it TLS now.

But the problem didn’t exist in the same way with SSL/TLS: SSL was actively killed off in favour of TLS so that regardless of what you call it, you’re actually dealing with TLS. But with feeds, RSS is still supported, and so the mindshare problem happens: people hear about RSS and so implement the inferior and problematic RSS rather than Atom, because they’ve never even heard of Atom and so don’t know that it’s what they should have implemented instead, because it’s more robust. Calling feeds RSS perpetuates RSS, which is undesirable.

I say just advertise it as “feeds”, not “RSS feeds”. Atom is an implementation detail, just like RSS should be. (Well, RSS should be dead in general feeds, with only podcast feeds keeping it alive.)

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#25

Post by Vinnl »

chrismorgan wrote: post #14
Vinnl wrote: post #12

Earlier quoted context omitted.

Ah, you mean the ones inside the contents? That's a good one. I'm not sure if that's easily fixable, but I'll give it some thought. For those interested, I'll track it here: https://gitlab.com/vincenttunru/feed-me-up-scotty/-/issues/1

The appropriate way is to use the xml:base attribute, as demonstrated in the example at https://datatracker.ietf.org/doc/html/rfc4287#page-4 .

Thanks! I'll look at that.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#28

Post by toastal »

spicybright wrote: post #2

I very simple plumbing tools like these.

This reminds me of the Soupault (https://soupault.app/) philosophy for building static sites. You write it in any language you want like, pass it trough Pandoc or AsciiDoctor as preprocessor, and postprocess with Lua and CSS selectors.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#29

Post by chrismorgan »

Vinnl wrote: post #25
chrismorgan wrote: post #14

Earlier quoted context omitted.

The appropriate way is to use the xml:base attribute, as demonstrated in the example at https://datatracker.ietf.org/doc/html/rfc4287#page-4 .

Thanks! I'll look at that.

> (Of course, for the combined feed this would be problematic.)

Not so; unlike the HTML element which applies to a document, the xml:base attribute is applied to an element and its descendants. The typical pattern (as shown in the RFC 4287 example) is to put it on each entry’s . In your markup, you’ll end up with each entry having its URL in three places:

  http://example.com/item
  
  …
Post reply on HN