Show HN: RSS feeds for arbitrary websites using CSS selectors • Hacker News Board

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#41

Post by ricardo81 » Mon, Jul 05, 2021, 6:31 PM UTC

bscphil wrote: post #38

ricardo81 wrote: post #30

Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.

I'm not sure I understand the criticism. What exactly is supposed to be the alternative? It's not like this is supposed to be used for high-availability purposes or stock trading or something. It scratches a particular itch for a handful of end users who just want to get all their news in their feed reader. It's expected that they'll maintain their own scraper parameters.

Pick a number of programming languages and you can scrape a page in 10-20 lines of code in many cases. The barrier to entry is understanding the DOM layout for a particular website, which is subject to change at any moment.

Purely IMO, a more friendly way to go about it to abstract from code and CSS knowledge is to run a UI that highlights elements, lets you select them, select the title, description, link etc and there you go. Same thing but without the knowledge of DOM/XPath/selectors.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#42

Post by eyedontgetit » Mon, Jul 05, 2021, 6:33 PM UTC

ricardo81 wrote: post #36

Vinnl wrote: post #35

Earlier quoted context omitted.

Ideally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…

Indeed, but the scraping workaround is brittle. We could make rules for all sites on the web that don't have a feed or API but it's not all that manageable. It also begs the question of why they don't make their content that way, perhaps it was by choice. Especially if you use this kind of tool to syndicate things. Not quite sure of the novelty of this one given that people have scraped things for decades. There's pr…

Re-stating the "brittleness" over and over doesn't add any weight to your argument. Regardless of whether or not the solution is brittle - it is still a viable solution for users who don't care or mind maintaining it. It sounds like you're not the target user and that's ok...

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#43

Post by kortilla » Mon, Jul 05, 2021, 6:35 PM UTC

ricardo81 wrote: post #30

Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.

Website doesn’t offer API. So we must scrape. This is a tool for that paradigm.

Your comment is “scraping is brittle”, which is not helpful. Everyone knows scraping sucks. It’s why these tools are being made to make it less unpleasant.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#44

Post by pmlnr » Mon, Jul 05, 2021, 6:35 PM UTC

chrismorgan wrote: post #18

I’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.

Oh, yes, please revive the bloody format wars /s

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#45

Post by ricardo81 » Mon, Jul 05, 2021, 6:37 PM UTC

kortilla wrote: post #43

ricardo81 wrote: post #30

Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.

Website doesn’t offer API. So we must scrape. This is a tool for that paradigm. Your comment is “scraping is brittle”, which is not helpful. Everyone knows scraping sucks. It’s why these tools are being made to make it less unpleasant.

It's not even that 'scraping sucks', that's just a stigma to the word. The point is that it's not stable and having to have DOM knowledge to select the paths is less inclusive.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#46

Post by Vinnl » Mon, Jul 05, 2021, 6:39 PM UTC

ricardo81 wrote: post #36

Vinnl wrote: post #35

Earlier quoted context omitted.

Ideally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…

Indeed, but the scraping workaround is brittle. We could make rules for all sites on the web that don't have a feed or API but it's not all that manageable. It also begs the question of why they don't make their content that way, perhaps it was by choice. Especially if you use this kind of tool to syndicate things. Not quite sure of the novelty of this one given that people have scraped things for decades. There's pr…

If you have a less brittle alternative that allows me to get the latest real estate listings in my area in my feed reader before it hits the aggregator sites, I'm all ears!

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#47

Post by moehm » Mon, Jul 05, 2021, 6:44 PM UTC

ricardo81 wrote: post #45

kortilla wrote: post #43

Earlier quoted context omitted.

Website doesn’t offer API. So we must scrape. This is a tool for that paradigm. Your comment is “scraping is brittle”, which is not helpful. Everyone knows scraping sucks. It’s why these tools are being made to make it less unpleasant.

It's not even that 'scraping sucks', that's just a stigma to the word. The point is that it's not stable and having to have DOM knowledge to select the paths is less inclusive.

> having to have DOM knowledge to select the paths is less inclusive

Right click on the element in Firefox, click "inspect element" and it shows you the unique selector for that element.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#48

Post by ricardo81 » Mon, Jul 05, 2021, 6:45 PM UTC

Vinnl wrote: post #35

ricardo81 wrote: post #34

Earlier quoted context omitted.

Not sure for the downvote reason. Like I say scraping is brittle. I've done hundreds of scraping projects in the past. A recent one was taking 50 shopping sites looking for what products they covered. Safe to say you look at it a month later and the XPaths/CSS selectors/whatever scraping method you used has changed. This is why APIs exist.

Ideally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…

I can't reply to the other subcomments because of the downvotes.

Just to be clear, I'm not against scraping. Done it myself plenty, point was that it's hard to maintain and if I were to make a tool for the masses, it'd be in a UI that highlights elements and lets you select them rather than dig around the DOM. Pretty sure things like this have existed for 10 years +

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#49

Post by Vinnl » Mon, Jul 05, 2021, 6:53 PM UTC

ricardo81 wrote: post #48

Vinnl wrote: post #35

Earlier quoted context omitted.

Ideally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…

I can't reply to the other subcomments because of the downvotes. Just to be clear, I'm not against scraping. Done it myself plenty, point was that it's hard to maintain and if I were to make a tool for the masses, it'd be in a UI that highlights elements and lets you select them rather than dig around the DOM. Pretty sure things like this have existed for 10 years +

Ah that's fine, this is explicitly not a tool for the masses. The problem I had with tools like the ones you describe are that either they just use string matching, or the selectors they generate are particularly brittle (e.g. incorporating randomised class names, like you mentioned elsewhere).

Given that there wasn't really an alternative to scraping for me, I wanted to at least be able to pick selectors myself that were less likely to break with minor changes. Then I figured there might be more people who know CSS and have the same desires, hence my sharing it here :)

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#50

Post by lhnz » Mon, Jul 05, 2021, 6:54 PM UTC

ricardo81 wrote: post #30

Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.

If you scrape using accessible parts of the page (e.g. `aria-` attributes) it is less likely to be brittle, since if it were it would mean their site had stopped being accessible to screen readers, etc.