Live data from Hacker News

Generate RSS feed for any website using CSS selectors

rss-bridge.org

41–50 of 55 posts

Re: Generate RSS feed for any website using CSS selectors

#41

Other services like this: https://www.fivefilters.org/feed-creator/

I created Feed Creator, so nice to see it mentioned in the comments :)

I've written two blog posts about how we go about using CSS selectors when working with Feed Creator. Might be useful for those looking to do the same with RSS-Bridge.

How to turn a webpage into an RSS feed using Feed Creator

Part 1: https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-...

Part 2 (using more advanced selectors): https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-...

Re: Generate RSS feed for any website using CSS selectors

#44

I've wondered why people have tried all sorts of cumbersome ways to splice metadata onto HTML like RDFa but never tried the obvious approach of basing extraction rules on CSS selectors... Often these work without the cooperation of the target site so long as they use CSS the way it was supposed be used (e.g. not tailwind, bootstrap, etc.)

Back in the optimistic 2000s there was the idea of GRDDL – using XSLT stylesheets and XPath selectors for extracting stuff, e.g. microformats, HTML meta, FOAF, etc: https://www.w3.org/TR/grddl/

Having learned xpath and a little xslt I've always wondered why it isn't more popular. It seems like a powerhouse for reading and transforming data from XML type documents. I've found it hard to find decent resources to learn more than the basics (and none for xquery) because of lack of popularity nowadays, but I do thing it's a skill you should have like SQL and regex. Seems a no brainer.

Re: Generate RSS feed for any website using CSS selectors

#45
post #37
post #28

What's the easiset way to also run a few basic filters on the site/RSS feed's content to make it truly shine vs simplistic scraping, like - splitting the full feed by theme of the article into separate feeds and at the same time - remove a few keywords and also - get article length and split into a long / short feed - Or maybe get what you used to have on some news sites - subscribe only to a specific author instead…

you could start by pushing all articles into a database; have another process quickly label/tag the entries based on the criteria you care about; web or tui app to show you only the entries you care about; slower clean up job for entries you don't care to keep around anymore

Thanks, but I meant which of the RSS services offers this basic filtering? From a dozen I know of, including paid ones, at most you get keywords black/white lists, which is too limiting Used to use Huginn for that on Heroku

Re: Generate RSS feed for any website using CSS selectors

#46

I run my own instance of RSS Bridge to keep track of authors that I like on Goodreads. It works pretty well, although every once in a while Goodreads hiccups, and then RSS bridge gives me a bunch of "new posts" that are actually error messages.

Hey, I wrote the Goodreads bridge for exactly this usecase. I’ll try to see if I can filter out the error messages.

Thanks! I've been meaning to play with the code and see if I could see if I could figure out how to add a few more features:

* Generate RSS feeds from book series

* Filter out translations

* Filter out compilations (not sure if this one is really plausible)

Any pointers on how I might accomplish some of those?

Re: Generate RSS feed for any website using CSS selectors

#47
post #17

RSSHub[0] is in the same ballpark, but consists of a large library of site-specific code[1][2]. [0] https://github.com/DIYgod/RSSHub/ [1] https://github.com/DIYgod/RSSHub/tree/master/lib/routes [2] https://github.com/DIYgod/RSSHub/tree/master/lib/v2

RSS Bridge also has a large library of site-specific code, CSS is just another of the hundred of solution they offer. And there are some other projects collecting and maintaining recipes for scrapping data from sites. Calibre for example and youtube-dl/yt-dlp for videos. Seeing so many projects doing all the same, I kinda feel sad that they are not cooperating to maintain a central recipe-collection.

Re: Generate RSS feed for any website using CSS selectors

#48

Earlier quoted context omitted.

Back in the optimistic 2000s there was the idea of GRDDL – using XSLT stylesheets and XPath selectors for extracting stuff, e.g. microformats, HTML meta, FOAF, etc: https://www.w3.org/TR/grddl/

Having learned xpath and a little xslt I've always wondered why it isn't more popular. It seems like a powerhouse for reading and transforming data from XML type documents. I've found it hard to find decent resources to learn more than the basics (and none for xquery) because of lack of popularity nowadays, but I do thing it's a skill you should have like SQL and regex. Seems a no brainer.

I’ve thought about that. My first take on XSLT was that it was “too complicated”, I got to talking with XSLT enthusiasts later and found out how many good ideas XSLT has in it.

My take is that some specifications can be written out in a linear way where you can start reading at the beginning and work to end and not feel like you need to read ahead.

Some specs have a minor discontinuity, I remember perceiving it in the K and R book on C but it seemed like there was just one kink in it and if you read the book twice you’d do OK.

Books in C++ are worse and have numerous topics that resist being put in the right order. It’s not unusual for “resource acquisition is initialization” to be repeated hundreds of times before it is defined, for instance.

That circularity is both a function of the domain and also a function of the text, I think a certain amount of circularity is inherent to many domains, but frequently you can bootstrap a domain by dividing it into numerous layers and put the circularity into a layer built just to manage the circularity.

XSLT, XMLSchema, and many XML specs have that kind of circular structure, you are left wondering what exact kind of machine is required to implement it so you can look at the spec and have a hard time understanding how to do easy things and no grasp of the hard-looking things that are actually easy. Couple that with numerous sharp edges in XML such as numeric values not being allowed in ID or IDREF fields (hate to break it to them but numeric identifiers are rampant in the jndustry) and it is no wonder people would rather use deeply lame ‘standards’ like JSON that lack comments, aren’t really clear about the semantics of numbers, and don’t have the moral authority to say “quit screwing around and just use ISO 8601 dares.

Now I finally realized the OWL spec is perfectly clear in the sense that you can understand what it really does by understanding the mapping of OWL axioms to first order logic, but the trouble is that logic is the most treacherous branch of mathematics.

Re: Generate RSS feed for any website using CSS selectors

#49
post #28

What's the easiset way to also run a few basic filters on the site/RSS feed's content to make it truly shine vs simplistic scraping, like - splitting the full feed by theme of the article into separate feeds and at the same time - remove a few keywords and also - get article length and split into a long / short feed - Or maybe get what you used to have on some news sites - subscribe only to a specific author instead…

Write a parser for rss-bridge that takes a rss feed in, does what you need, and spits a feed out

I don't know any service that does that automatically but it's attainable to have a generic way of doing what you need. That's the power of rss-bridge: make the feed you want from content that already exists

Re: Generate RSS feed for any website using CSS selectors

#50
post #38

Earlier quoted context omitted.

And it also became harder due to people thinking random ids and class names are totally fine. Super annoyed by that. It feels like they are forcing their vision onto the user, while the user does not want their vision and could not care less.

The web was nicer when you could inspect, learn, & riff off of what others where doing in the industry–like the old music industry used to do when covering & borrowing a phrase was considered homage not grounds for lawsuit. It’s now all meant to be closed off & behind build tools that complect the output where most folks don’t even know how their pipeline works; and this is strange since the simple tools of HTML, CSS…

Agree completely. As a minor nicety, though, I’ve noticed most sites publish sourcemaps in production now. So, in a few ways it’s become more possible to inspect and study JS, compared to when sourcemaps weren’t there, and you could only see mangled source.
Post reply on HN