Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a tool that watches webpages and exposes changes as RSS

sitespy.app

Post reply on HN
Print view View original

31–40 of 86 posts

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#31

Post by vkuprin »

ahmedfromtunis wrote: post #20

As a (former) reporter, site monitoring is a big part of what I do on a daily basis and I used many, many such services. I can attest that, at least from the landing page, this seems to be a very good execution of the concept, especially the text-based diffing to easily spot what changed and, most importantly, how. The biggest hurdle for such apps however are 'js-based browser-rendered sites' or whatever they're call…

Thanks, that’s a really good question. Site Spy uses a real browser flow, so it generally handles JS-rendered pages much better than simple HTML-only polling tools. In practice, the trickier cases tend to be sites with aggressive anti-bot protection or messy login/session flows rather than JS itself. I’m trying to make those limitations clearer so people don’t just hit a vague failure and feel let down

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#33

Post by vkuprin »

nicbou wrote: post #22

Buddy I love you! I have wanted this for so long! My job relies on following many German laws, bureaucracy pages and the like. In the long run I want specific changes on external pages to trigger pull requests in my code (e.g. to update a tax threshold). This requires building blocks that don't exist, and that I can't find time to code and maintain myself. I currently use Wachete, but since over a year, it triggers r…

German laws and bureaucracy pages are exactly the kind of thing where tracking one specific part of a page is much more useful than watching the whole page. And yeah, more control over check frequency makes a lot of sense if monthly checks are enough and rate limits are the main problem. I’d be curious what kind of schedule would work best for you there?

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#34

Post by vkuprin »

layman51 wrote: post #26

How might this tool work in terms of “archiving” a site? This is just something I was wondering given the recent change and controversy about archiving service sites on Wikipedia.

Site Spy keeps snapshot history, so you can revisit older versions of a page and inspect how it changed over time, not just get the latest alert. I’d describe it more as monitoring with retained history than as a dedicated public archive, but deeper archival integrations are definitely something I’ve thought about

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#35

Post by plutokras »

I have my own hobby RSS server built around the Google Reader API. Two of my plugins are pretty similar to what you described: one checks a page’s current state against the last saved version and publishes an entry if anything changed, the other is basically a CSS selector-based feed builder. Always good to see RSS content here, thanks for posting!

On your questions: some people prefer RSS, others email, and services exist to convert between the two in both directions. My own rule of thumb is email for things that need actual attention and RSS for everything that can wait. If you’re thinking about turning this into a service, supporting both would make sense since people are pretty split on this.

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#36

Post by Hauk307 »

This is cool. I'd use it to track when state wildlife agencies update their regulation pages — those change once a year with no announcement and I always miss it. Element-level tracking would be perfect for that vs watching the whole page. To answer your question: I'd want both RSS and direct alerts (email/push) depending on urgency.

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#37

Post by deceptionatd »

hinkley wrote: post #15

Back in 2000 I worked for a company that was trying to turn something like this into the foundation for a search engine. Essentially instead of having a bunch of search engines and AI spamming your site, the idea was that they would get a feed. You would essentially scan your own website. As crawlers grew from an occasional visitor to an actual problem (an inordinate percent of all consumer traffic at the SaaS I work…

Well-designed sitemaps and use of something like https://www.indexnow.org/ helps.

Cloudflare has Crawler Hints which works well IME: https://blog.cloudflare.com/crawler-hints-how-cloudflare-is-...

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#38

Post by nicbou »

vkuprin wrote: post #33
nicbou wrote: post #22

Buddy I love you! I have wanted this for so long! My job relies on following many German laws, bureaucracy pages and the like. In the long run I want specific changes on external pages to trigger pull requests in my code (e.g. to update a tax threshold). This requires building blocks that don't exist, and that I can't find time to code and maintain myself. I currently use Wachete, but since over a year, it triggers r…

German laws and bureaucracy pages are exactly the kind of thing where tracking one specific part of a page is much more useful than watching the whole page. And yeah, more control over check frequency makes a lot of sense if monthly checks are enough and rate limits are the main problem. I’d be curious what kind of schedule would work best for you there?

Monthly is fine, but not monthly all at once, because I watch multiple pages on one website, and that triggers the rate limiting.

The ideal pipeline for me would be "notice a change in a specific part of a page, use a very small LLM to extract a value or answer a question, update a constant in a file and make a pull request".

I've been thinking about this pipeline for a long time because my work depends on it, but nothing like it seems to exist yet. I'll probably write my own, but I just can't find the time.

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#39

Post by nicbou »

beepbooptheory wrote: post #16
xnx wrote: post #5

I like https://github.com/dgtlmoon/changedetection.io for this. Open source and free to run locally or use their Saas service.

Sure but this one has a MCP server, costs money, and was presumably made last night!

It's been around for a while and recommended by many. I tried it myself and it's okay.

Re: Show HN: I built a tool that watches webpages and exposes changes as RSS

#40

Post by OSaMaBiNLoGiN »

Tool looks useful. But how is it that toggling between light/dark mode results in a multi-second freeze..? Scrolling drops frames, confirmed with dev tools.

Tested on m1 pro 2021 laptop and recent higher-end (4080, 14700k, etc) desktop. Same on both.

The fuck?

Post reply on HN