Show HN: RSS feeds for arbitrary websites using CSS selectors • Hacker News Board

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#81

Post by mhitza » Mon, Jul 05, 2021, 9:23 PM UTC

This could nicely supplement my GitHub automation that emails feed digests https://github.com/mhitza/subscriptions-digest

Similarly to my repository, I think I would suggest the option to fetch the configuration file from an external resource defined via an action secret. For my automation I'm using a Gist (not sure if Gitlab has same thing; also private but publicly accessible snippets).

At least that way you can keep your own feed configuration while allowing those that fork the repository to not have to manually fix conflicts within the feeds.toml config.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#82

Post by DoreenMichele » Mon, Jul 05, 2021, 9:27 PM UTC

KirillPanov wrote: post #80

lhnz wrote: post #50

Earlier quoted context omitted.

If you scrape using accessible parts of the page (e.g. `aria-` attributes) it is less likely to be brittle, since if it were it would mean their site had stopped being accessible to screen readers, etc.

Why the downvotes here? This is objectively true and, in my experience, very very useful. Screen readers are scrapers too. "Scraper" does not mean "user agent I don't like".

Is there someplace I can read more about this idea? Any aspect of it. I'm one of the less technically literate members of HN, but I'm a co-owner of a blind dev group.

Please and thank you.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#83

Post by KirillPanov » Mon, Jul 05, 2021, 9:32 PM UTC

DoreenMichele wrote: post #82

KirillPanov wrote: post #80

Earlier quoted context omitted.

Why the downvotes here? This is objectively true and, in my experience, very very useful. Screen readers are scrapers too. "Scraper" does not mean "user agent I don't like".

Is there someplace I can read more about this idea? Any aspect of it. I'm one of the less technically literate members of HN, but I'm a co-owner of a blind dev group. Please and thank you.

https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#84

Post by [deleted] » Mon, Jul 05, 2021, 9:41 PM UTC

KirillPanov wrote: post #83

DoreenMichele wrote: post #82

Earlier quoted context omitted.

Is there someplace I can read more about this idea? Any aspect of it. I'm one of the less technically literate members of HN, but I'm a co-owner of a blind dev group. Please and thank you.

https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...

[deleted]

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#85

Post by onli » Mon, Jul 05, 2021, 9:49 PM UTC

ricardo81 wrote: post #72

detaro wrote: post #61

Earlier quoted context omitted.

who has claimed there is something fundamentally new here?

Presenting something on the home page of HN would suggest something novel. I must be missing the point because of the downvotes. Happy to be enlightened. Pick CSS selectors and scrape something? That's quintessentially scraping.

I'd say the novel part of the project is the Github actions/Gitlab CI integration. I haven't seen that used for RSS generation yet. That the selectors are stored in a config file is also unusual for these types of feed generation projects.

Though Show HNs do not need to be novel to get to the top. And RSS related projects seem to go to the frontpage simply because of the fondness HN has for the technology. Also, don't forget https://xkcd.com/1053/ :)

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#86

Post by kwerk » Mon, Jul 05, 2021, 10:26 PM UTC

lorey wrote: post #27

In case anyone wants to detect the selectors automatically, here's a small python library I wrote that does it for you: https://github.com/lorey/mlscraper

This looks interesting! Will you be adding an open source license?

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#87

Post by markdown » Mon, Jul 05, 2021, 10:47 PM UTC

Vinnl wrote: post #70

markdown wrote: post #64

Great project, but that's a shitty name.

Heh, one thing I've learned over the years is that a great way to not actually publish my side projects is to overthink the name, so nowadays I just pick one and go with it when I'm ready to publish it.

I can't disagree with that :)

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#88

Post by kdbg » Mon, Jul 05, 2021, 10:57 PM UTC

Kinda on a related note I found myself needing to make a bunch of these sorts of scraped feeds. The problem for me was the lack of date parsing support which I sorely needed (and it doesn't appear like this option supports it either)

I ended up writing my own CLI tool that similarly supports CSS selectors for feed generation: https://github.com/dayzerosec/feedgen

I did write it specifically for my use-case so there are some "warts" on it like custom generators for HackerOne and Google's Monorail bug tracker. But perhaps someone else might benefit from its ability to create slightly more complicated RSS, Atom, or JSON feeds.

Example config with date parsing: https://github.com/dayzerosec/feedgen/blob/main/configs/bish...

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#89

Post by peanutz454 » Mon, Jul 05, 2021, 11:10 PM UTC

On a job website I wondered if I could use CSS selectors to get to the important stuff. I think a lot of people have had those thoughts, and the website knew. While the page might have some repeatable structure, the ids and classes were just randomly generated. Also, After you open the job listing, they too were all different because the job poster had the ability to set up their own mini html that was embedded in the page.

Re: Show HN: RSS feeds for arbitrary websites using CSS selectors

#90

Post by VoidWhisperer » Mon, Jul 05, 2021, 11:17 PM UTC

peanutz454 wrote: post #89

On a job website I wondered if I could use CSS selectors to get to the important stuff. I think a lot of people have had those thoughts, and the website knew. While the page might have some repeatable structure, the ids and classes were just randomly generated. Also, After you open the job listing, they too were all different because the job poster had the ability to set up their own mini html that was embedded in th…

It is entirely possible that they did this to prevent scraping, but it is also possible it isn't intentional like that - for example, if you use the 'styled components' react library, it ends up spitting out a ton of class names that are basically gibberish, not intentionally.