Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Changedetection.io detect changes in websites and JSON feeds

github.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#21

Post by biggunz »

andkon wrote: post #19
derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

I think you're right that the services themselves break, but more often than not, the failure mode is that what page you're monitoring changes its structure, and then every refresh gives you data that is incommensurate with what came before. It's why we version APIs, but there's no such versioning we expose on websites. I think that also points to why what you propose would be very hard to do: in HTML pages, even if…

"But I think the meaning of the change is important" this is exactly right, is it a whitespace? is it the text? what is "change" ?

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#22

Post by biggunz »

CharlesW wrote: post #14
hackersnewses wrote: post #8

Earlier quoted context omitted.

> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).

> You're looking for Zyte (formerly ScrapingHub). Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.

changedetection.io supports filters and triggers which all can result in a notification to a JSON call or anyone of the existing hundreds of other services (email, discord, etc)

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#23

Post by biggunz »

layer8 wrote: post #13
derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.

Self-hosting is the only safest bet, or share a real server somewhere with some friends, or however you like

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#24

Post by biggunz »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

I've been running this software locally as a docker container on my laptop since about February 2021, obviously with a few updates/restarts, but generally it is set and forget

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#28

Post by [deleted] »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

[deleted]

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#29

Post by jimmydorry »

Nice little project. I played around with it for a bit, but couldn't figure out how to extract snippets (e.g. a list of headlines, like from the hackernews home page). It seems to be good at grabbing a dumb dump of text within a given CSS selector, but no regex pattern matching within a line (the regex fields appear to toss out all lines that match the pattern).
Post reply on HN