simple, clean and useful. I'm trying it out. Good job!
Show HN: Sleeping Spider – Web page change monitor
11–20 of 42 posts
Re: Show HN: Sleeping Spider – Web page change monitor
#12Re: Show HN: Sleeping Spider – Web page change monitor
#13Does this site respect robots.txt?
Does it render javascript content (so e.g. angular sites work)?
Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes?
Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change?
Will I get a notification if a page disappears entirely?
Does it scrape only once per day or more often and alert only once a day?
What user-agent does it scrape from on the off-chance someone wants to block it?
Does it diff binary data.. for example, if I ask it to watch an image url will it hash the image and notify me when it gets updated?
Does it auto-detect pages with rss feeds or pubsub feeds and subscribe to them?
Does it understand twitter so if I watch a user, it only alerts me of tweets, not e.g. a change in the number of followers?
Does it have special code for any website whatsoever?
Is the webpage parsing/diffing library / algorithm open source somewhere?
Is there a monitization plan, and if not how can I trust it won't vanish?
Is there a limit on the number of sites I can watch?
I like the simplistic design, but it would be nice if there were more information.
Re: Show HN: Sleeping Spider – Web page change monitor
#14Does this site respect robots.txt? Does it render javascript content (so e.g. angular sites work)? Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change? Will I get a notification if a page disappears enti…
Go write your own then.
I find this a bizarre response to a simple and easy to understand tool.
Re: Show HN: Sleeping Spider – Web page change monitor
#15Does this site respect robots.txt? Does it render javascript content (so e.g. angular sites work)? Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change? Will I get a notification if a page disappears enti…
Go write your own then. I find this a bizarre response to a simple and easy to understand tool.
If it's easy to understand then I guess you can answer my questions?
I don't think "go write your own" is a legitimate response to my questions (if that's ever a legitimate response).
Re: Show HN: Sleeping Spider – Web page change monitor
#16Earlier quoted context omitted.
Go write your own then. I find this a bizarre response to a simple and easy to understand tool.
If it's easy to understand then I guess you can answer my questions? I don't think "go write your own" is a legitimate response to my questions (if that's ever a legitimate response).
I'm using websec as an existing tool to watch website changes. It's not perfect but it's good enough.
For a while I had a webapp to do a better job but it never got any traction, most likely because I had no idea how to get traction. It's an interesting exercise to write such a tool though.
Re: Show HN: Sleeping Spider – Web page change monitor
#17If you prefer privacy and control over your own tools, there are the FLOSS http://specto.sourceforge.net/ or the free(as in no cost)ware http://www.cmcode.co.uk/webmon/ for Windows
Re: Show HN: Sleeping Spider – Web page change monitor
#18Does this site respect robots.txt? Does it render javascript content (so e.g. angular sites work)? Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change? Will I get a notification if a page disappears enti…
Go write your own then. I find this a bizarre response to a simple and easy to understand tool.
I thought that at least part of Show HN was to gather feedback and look into how things are implemented, which features are supported / unsupported etc.
Therefore, I personally find this list of questions very relevant. Some of those questions I was asking myself when looking at the page. The fact that this information wasn't clear from the website itself is also important. At least some of the users of such tool would most likely want to know what to expect.
Perhaps it could have been asked in a more positive way (not entirely sure how, but maybe explaining why those things matter), but the questions themselves are important and I'd be curious to know the answers.
Re: Show HN: Sleeping Spider – Web page change monitor
#19Does this site respect robots.txt? Does it render javascript content (so e.g. angular sites work)? Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change? Will I get a notification if a page disappears enti…
Go write your own then. I find this a bizarre response to a simple and easy to understand tool.
Your post is a bizarre response to a simple list of questions. And the "go write your own then" is kind of offensive.
Re: Show HN: Sleeping Spider – Web page change monitor
#20Adding watch results in a white screen of nothing, no new watches added. I'm trying to monitor the Nexus 6 page :D https://play.google.com/store/devices/details/Nexus_6_32GB_C... For my use case, anything more than an hour delay wouldn't be practical. Ideally it would be up to the minute, but I'd imagine that would cause other issues for a service like this. In the past I just used a Chrome Extension. But living in V…
The probe received a 403 forbidden error from google. I realized that google store forbid crawls. So a simple http request won't work.
I was not aware of this situation before. I will see what I can do.