Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Changedetection.io detect changes in websites and JSON feeds

github.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#11

Post by iamstupidsimple »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

StackDriver supports health checks: https://cloud.google.com/monitoring/uptime-checks

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#13

Post by layer8 »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#14

Post by CharlesW »

hackersnewses wrote: post #8
derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).

> You're looking for Zyte (formerly ScrapingHub).

Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#15

Post by biggunz »

TheKnack wrote: post #12

I used this to replace Visualping.io. The only features it lacks compared to VisualPing is a GUI for selecting page elements to monitor and fully rendered diff results. It either monitors the whole page or you can add a CSS selector manually.

It accepts CSS filters - click on EDIT, Then "Filters & Triggers" and there you can fill in a CSS filter, more stuff coming soon :)

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#16

Post by biggunz »

olex wrote: post #9

Nice and easy to self-host with Docker, thanks. Might save me more than a few bookmarks that I check every couple weeks manually. Small feature wish: optionally fetch and display favicons next to the URLs or s in the website list.

Great idea!

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#18

Post by biggunz »

s-xyz wrote: post #17

How did you create the screenshot difference? https://raw.githubusercontent.com/dgtlmoon/changedetection.i...

Just wait for a change to be detected and then clicked "DIFF" button in the main list, easy.. the screenshot is just from my browser of my local installation of the software

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#19

Post by andkon »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

I think you're right that the services themselves break, but more often than not, the failure mode is that what page you're monitoring changes its structure, and then every refresh gives you data that is incommensurate with what came before. It's why we version APIs, but there's no such versioning we expose on websites.

I think that also points to why what you propose would be very hard to do: in HTML pages, even if the page structure (a new div level) changes, it's hard to say if that's actually a fundamental change to how they've architected the information hierarchy you're trying to get notified out, or if that is indeed the information itself. It's ambiguous.

Maybe that's okay? Like, if all you want is a notification about every time a thing changes, regardless of the meaning of the change, then what you're describing is possible. But I think the meaning of the change is important, and that's ultimately what shifts over time in ways computers cannot grok.

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#20

Post by biggunz »

eterps wrote: post #3

Nice! Does it have an API for adding URLs to monitor? Would be nice to have a browser extension to do that easily.

Yeah, coming in the next week or so actually, or feel free to open a PR! also adding an API to retrieve snapshots (text/JSON) too
Post reply on HN