Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Changedetection.io detect changes in websites and JSON feeds

github.com

Post reply on HN
Print view View original

1–10 of 33 posts

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#5

Post by derefr »

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them.

For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their scraping algorithms in ways that make previously-working configs break; etc.

Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers?

I suppose I could always find a self-hosted software product for this, and deploy it myself. But though the software would be stable, that's not very "set and forget" either; I'd be on the hook for applying updates, etc.

I want something that is to the "HTTP response change detection" abstraction, as S3-alike object-lifecycle event notifications are to the "object-storage change detection" abstraction. Some simple plumbing primitive that does exactly one thing, exactly one (obvious) way, that can be used to build the higher-level abstractions I want, with nine nines of uptime behind it, and the expectation it'll stick around for at least the next decade.

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#6

Post by blacksmith_tb »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

So you just need to set up two of these services, and have each monitor the other... Have to agree, it does feel like a case where rolling your own may be less elegant, but at least you'll know how the plumbing works (or if it doesn't).

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#7

Post by toomuchtodo »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

Feels like you need something like a non profit providing this service, similar to Let's Encrypt, with enough funds to keep a two person team (bus factor) compensated for maintenance and ongoing support.

Re: Show HN: Changedetection.io detect changes in websites and JSON feeds

#8

Post by hackersnewses »

derefr wrote: post #5

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their sc…

> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers?

You're looking for Zyte (formerly ScrapingHub).

Post reply on HN