Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Checkbot for Chrome – web crawler that tests for web best practices

checkbot.io

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#41

Post by scrollaway »

seanwilson wrote: post #39
scrollaway wrote: post #37

Earlier quoted context omitted.

> Would being able to set cookies or sending custom headers help? I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding…

> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and thi…

> Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this.

I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain.

Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, skip the url parameters, etc) and 2. let me add that pattern after the urls have already been crawled (and make sure I can see which URLs are affected).

Keep in mind that this is pretty raw feedback and you know your product better than me, but I definitely don't think the URLs to ignore is usable right now.

I'm off to bed, I hope all that helped. :)

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#42

Post by seanwilson »

scrollaway wrote: post #41
seanwilson wrote: post #39

Earlier quoted context omitted.

> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and thi…

> Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain. Re URLs to ignore: Ok, I see, that wasn't clear though. My two-fold suggestion is 1. Add a way to specify plain loose matching (eg. just /account/login/, s…

> I don't. I'm not sure how Google picks up on the fact that they're login walls. Maybe it's heuristics? Someone better at SEO than me could explain.

Google is likely following the redirect to the login page and seeing that the login page text isn't relevant to your search results at a guess. It's not a big deal if Google hides those pages but in Checkbot perhaps those are pages you want to examine so I'll need to think about what to do here.

> Keep in mind that this is pretty raw feedback and you know your product better than me

This kind of feedback is amazing so keep it coming! Knowing the first thing you thought before taking the time to fully investigate a feature is super useful because most users would be gone already if they were confused.

> but I definitely don't think the URLs to ignore is usable right now.

Yes, fully agree with that. I think if you're dealing with regexes, you want a way to test them as it's too easy to make a mistake.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#43

Post by PudgePacket »

pimlottc wrote: post #30
PudgePacket wrote: post #27

Earlier quoted context omitted.

I've never been able to use that, every time I run the test it says initial page load is 10s, but I'm running it against a static website hosted locally.. how is it taking so long? Even if I visited my site from the other side of the world over 3g it would be faster than lighthouse accessing it locally.

Part of the audit includes throttling your connection to simulate a device on a slower link. https://github.com/GoogleChrome/lighthouse/blob/master/docs/...

Still couldn't account for 10s for 5kb of html, something must have been really wrong, maybe I configured it incorrectly.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#44

Post by acct1771 »

keithnz wrote: post #25
gerdesj wrote: post #21

Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems. First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its…

I agree, it runs nicely, and is nicely presented. The term best practice does irk me. But weirdly enough I think the use of that term is why I decided to give it a go.

Love-hate relationship with marketing is probably pretty common in this crowd!

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#46

Post by seanwilson »

queezey wrote: post #8

Very cool. I noticed that it's mangling some of my URLs, though. `/!0ead1aEq` is getting turned into `/%210ead1aEq` (the exclamation point is getting percent-encoded), which leads to a bunch of spurious 404 errors. https://tools.ietf.org/html/rfc3986#section-3.3

Hmm, so when crawling, URLs are normalised and the URL library I'm using is normalising the "!" to "%21". Could you send me a working URL to test on? My email is sw@seanw.org if you want to use that.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#47

Post by mkorsak »

Seems really cool, I like this. One issue I'm having so far though is that after crawling my site it found one 404, and the link is (my domain)/page-not-found-test. It also says there's 0 inlinks to it, so I have no idea where it's getting the idea for this page from. It doesn't exist, but I've never linked to anything like it?
Post reply on HN