Show HN: RSS feeds for arbitrary websites using CSS selectors • Hacker News Board

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#51

Post by ricardo81 » Mon, Jul 05, 2021, 6:54 PM UTC

moehm wrote: post #47

ricardo81 wrote: post #45

Earlier quoted context omitted.

It's not even that 'scraping sucks', that's just a stigma to the word. The point is that it's not stable and having to have DOM knowledge to select the paths is less inclusive.

> having to have DOM knowledge to select the paths is less inclusive Right click on the element in Firefox, click "inspect element" and it shows you the unique selector for that element.

Are you aware that some websites use randomised css ids and classes? Have to wonder why. Google search results being an example.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#52

Post by ricardo81 » Mon, Jul 05, 2021, 6:59 PM UTC

Vinnl wrote: post #49

ricardo81 wrote: post #48

Earlier quoted context omitted.

I can't reply to the other subcomments because of the downvotes. Just to be clear, I'm not against scraping. Done it myself plenty, point was that it's hard to maintain and if I were to make a tool for the masses, it'd be in a UI that highlights elements and lets you select them rather than dig around the DOM. Pretty sure things like this have existed for 10 years +

Ah that's fine, this is explicitly not a tool for the masses. The problem I had with tools like the ones you describe are that either they just use string matching, or the selectors they generate are particularly brittle (e.g. incorporating randomised class names, like you mentioned elsewhere). Given that there wasn't really an alternative to scraping for me, I wanted to at least be able to pick selectors myself that…

I do have some code lying around somewhere that would revive wordpress blogs via wayback, using a UI where you could select elements in the 'theme' and restore the backend database. It's simply a messy business IMO. But for one off jobs, scraping is the go to.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#53

Post by playpause » Mon, Jul 05, 2021, 7:09 PM UTC

ricardo81 wrote: post #51

moehm wrote: post #47

Earlier quoted context omitted.

> having to have DOM knowledge to select the paths is less inclusive Right click on the element in Firefox, click "inspect element" and it shows you the unique selector for that element.

Are you aware that some websites use randomised css ids and classes? Have to wonder why. Google search results being an example.

There’s usually some way to target what you want through descendent/sibling, tag name, and attribute selectors.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#54

Post by moehm » Mon, Jul 05, 2021, 7:10 PM UTC

ricardo81 wrote: post #51

moehm wrote: post #47

Earlier quoted context omitted.

> having to have DOM knowledge to select the paths is less inclusive Right click on the element in Firefox, click "inspect element" and it shows you the unique selector for that element.

Are you aware that some websites use randomised css ids and classes? Have to wonder why. Google search results being an example.

Yes, I am. And this project is obviously not designed to handle that. (It can't even do pagination.)

But it can bring feed capabilities to simple, timeline like sites. Sites like most frameworks produce. And with the dev tools you can quickly find the needed dom path. It's limited, but easy to use. If it doesn't work, you need a real scraper which is an order of magnitude more complex. (I maintain some of them as well.)

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#55

Post by canada_dry » Mon, Jul 05, 2021, 7:11 PM UTC

Browsing OP's repositories was this gem: https://gitlab.com/vincenttunru/flatuscode A VSCode extension. That adds farts on every keypress. That's all.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#56

Post by hallway_monitor » Mon, Jul 05, 2021, 7:16 PM UTC

canada_dry wrote: post #55

Browsing OP's repositories was this gem: https://gitlab.com/vincenttunru/flatuscode A VSCode extension. That adds farts on every keypress. That's all.

Pretty great for April 1st... Or whenever you find an unlocked machine!

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#57

Post by alanchen » Mon, Jul 05, 2021, 7:17 PM UTC

This perhaps has more flexibility and can deal with almost any website.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#58

Post by ricardo81 » Mon, Jul 05, 2021, 7:18 PM UTC

playpause wrote: post #53

ricardo81 wrote: post #51

Earlier quoted context omitted.

Are you aware that some websites use randomised css ids and classes? Have to wonder why. Google search results being an example.

There’s usually some way to target what you want through descendent/sibling, tag name, and attribute selectors.

Yes, if you're determined enough you can scrape anything. Just not sure what the new thing here is.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#59

Post by canada_dry » Mon, Jul 05, 2021, 7:23 PM UTC

hallway_monitor wrote: post #56

canada_dry wrote: post #55

Browsing OP's repositories was this gem: https://gitlab.com/vincenttunru/flatuscode A VSCode extension. That adds farts on every keypress. That's all.

Pretty great for April 1st... Or whenever you find an unlocked machine!

When I saw it, I did do the quick calc based on the last update: worked out to around Christmas. OP might have just been bored over the holidays.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#60

Post by spacec0wb0y » Mon, Jul 05, 2021, 7:25 PM UTC

What do I do to get this working?

So far I've forked feeds, edited feeds.toml, checked it out as a branch gh-pages, pushed the branch up to github.

I can see the page at .github.io/feeds/" rel="nofollow">https://.github.io/feeds/ but .github.io/feeds/actions" rel="nofollow">https://.github.io/feeds/actions is just a 404.