Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generate RSS feed for any website using CSS selectors

rss-bridge.org

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Generate RSS feed for any website using CSS selectors

#21

Post by ttepasse »

PaulHoule wrote: post #8

I've wondered why people have tried all sorts of cumbersome ways to splice metadata onto HTML like RDFa but never tried the obvious approach of basing extraction rules on CSS selectors... Often these work without the cooperation of the target site so long as they use CSS the way it was supposed be used (e.g. not tailwind, bootstrap, etc.)

Back in the optimistic 2000s there was the idea of GRDDL – using XSLT stylesheets and XPath selectors for extracting stuff, e.g. microformats, HTML meta, FOAF, etc:

https://www.w3.org/TR/grddl/

Re: Generate RSS feed for any website using CSS selectors

#23

Post by 1vuio0pswjnm7 »

"Generate RSS feed for any website using CSS selectors"

For me, "CSS selectors" always seems like a deceptive term, if it means selecting HTML tag elements. What if the website does not use styling.

I read 1000s of websites, including all HN submissions, without using CSS. When I want to extract information from a website, I focus on patterns in the page. They might be HTML, they might be style elements, but they could be anything. I never assume that all websites will wrap the information I want in certain elements. There is a ridiculous amount of random variation amongst websites.

Re: Generate RSS feed for any website using CSS selectors

#24

Post by mmcwilliams »

1vuio0pswjnm7 wrote: post #23

"Generate RSS feed for any website using CSS selectors" For me, "CSS selectors" always seems like a deceptive term, if it means selecting HTML tag elements. What if the website does not use styling. I read 1000s of websites, including all HN submissions, without using CSS. When I want to extract information from a website, I focus on patterns in the page. They might be HTML, they might be style elements, but they cou…

I'm not sure that CSS being used on the page is a requirement. In the way that `h1 a` would be a valid CSS selector, in this case, would not be require that it be styled by a style sheet.

The key here is that it uses selectors, not the style sheets themselves.

Re: Generate RSS feed for any website using CSS selectors

#25

Post by daniel-s »

1vuio0pswjnm7 wrote: post #23

"Generate RSS feed for any website using CSS selectors" For me, "CSS selectors" always seems like a deceptive term, if it means selecting HTML tag elements. What if the website does not use styling. I read 1000s of websites, including all HN submissions, without using CSS. When I want to extract information from a website, I focus on patterns in the page. They might be HTML, they might be style elements, but they cou…

You just need to use the same logic, syntax as CSS' selectors to pick out can ntent from the page. That's something a little different to CSS to style.

Re: Generate RSS feed for any website using CSS selectors

#27

Post by notadev »

dagurp wrote: post #18

These days I just let chagpt generate a script that scrapes a site and spits out an rss file. Then I run it with cron.

I’m guessing they paste a portion of the website’s source then tell ChatGPT to generate a script that can generate an RSS feed from that site.

Re: Generate RSS feed for any website using CSS selectors

#28

Post by eviks »

What's the easiset way to also run a few basic filters on the site/RSS feed's content to make it truly shine vs simplistic scraping, like

- splitting the full feed by theme of the article into separate feeds and at the same time

- remove a few keywords and also

- get article length and split into a long / short feed

- Or maybe get what you used to have on some news sites - subscribe only to a specific author instead of getting bombarded with hundreds of items in a feed

Re: Generate RSS feed for any website using CSS selectors

#29

Post by captn3m0 »

nfriedly wrote: post #22

I run my own instance of RSS Bridge to keep track of authors that I like on Goodreads. It works pretty well, although every once in a while Goodreads hiccups, and then RSS bridge gives me a bunch of "new posts" that are actually error messages.

Hey, I wrote the Goodreads bridge for exactly this usecase. I’ll try to see if I can filter out the error messages.

Re: Generate RSS feed for any website using CSS selectors

#30

Post by captn3m0 »

bubblematrix wrote: post #19

This honestly is standard web scraping but these projects always catch my attention. You're bound at the mercy of rate-limiting firewalls (so you'll have to rotate proxies if you intend on using this heavily) on top of the standard CloudFront bot detection recaptcha, and div-obfuscation (a good example of this is Facebook).

rss-Bridge has decent caching support, customisable on a bridge level, so that comes pre-tuned and works well at low volumes for personal use.

At large scale, like the kind of traffic I started seeing when I ran a public rss-bridge Instagram/Telegram bridge - rate limits are unavoidable.

Post reply on HN