Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Checkbot for Chrome – web crawler that tests for web best practices

checkbot.io

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#31

Post by scrollaway »

seanwilson wrote: post #29
scrollaway wrote: post #22

Really cool stuff. Here's some initial feedback: "Avoid internal link redirects" -> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc -> Same problem as above with login walls ("Sign in to ..."). I get…

Awesome, thanks for the detailed feedback! As you can probably imagine, tweaking the rules to work with every imaginable website configuration is an ongoing process so this is super helpful. > All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. > "Use unique titles" / "Set page descriptions" / "Avoid URL para…

> Allowing Checkbot to login could help but I'll look into how to improve this.

Wouldn't help in my scenario FWIW, my site is oauth-only login.

> Hmm, any more suggestions on what to change here?

I move the copy link into a clipboard button next to the url (like Github's "clipboard" button next to URLs), and make the remainder of the links prominent buttons. I would also avoid taking users to a separate page when clicking any of them; rather, open a "sub view" below the url (eg a nested list).

If you have less-often-used actions, you could also add a "..." menu on the right side, or next to the buttons.

> Can you give more details here

Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#32

Post by wgjordan »

ratata wrote: post #15

Nice. very similar to Lighthouse https://developers.google.com/web/tools/lighthouse/#devtools

Similar feature-set at first glance, but not open-source, not free after beta ends, and created/maintained by an unknown solo developer. Nice site design, but I think the cards are pretty heavily stacked against this one.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#33

Post by seanwilson »

scrollaway wrote: post #31
seanwilson wrote: post #29

Earlier quoted context omitted.

Awesome, thanks for the detailed feedback! As you can probably imagine, tweaking the rules to work with every imaginable website configuration is an ongoing process so this is super helpful. > All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. > "Use unique titles" / "Set page descriptions" / "Avoid URL para…

> Allowing Checkbot to login could help but I'll look into how to improve this. Wouldn't help in my scenario FWIW, my site is oauth-only login. > Hmm, any more suggestions on what to change here? I move the copy link into a clipboard button next to the url (like Github's "clipboard" button next to URLs), and make the remainder of the links prominent buttons. I would also avoid taking users to a separate page when cli…

> Wouldn't help in my scenario FWIW, my site is oauth-only login.

Would being able to set cookies or sending custom headers help?

> Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it.

Hmm, so if you check "Explore" -> "External URLs" there's a ton of external links being checked for this URL. I'm not sure what you could do here except excluding hsreplay.net URLs from being checked.

Thanks for the other tips and examples, I'm actively working on this.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#34

Post by seanwilson »

wgjordan wrote: post #32
ratata wrote: post #15

Nice. very similar to Lighthouse https://developers.google.com/web/tools/lighthouse/#devtools

Similar feature-set at first glance, but not open-source, not free after beta ends, and created/maintained by an unknown solo developer. Nice site design, but I think the cards are pretty heavily stacked against this one.

The big difference is Checkbot crawls whole websites as opposed to checking a single page at a time. The Checkbot interface is designed around helping you hunt down issues that impact groups of pages and pages you didn't think to check. For example, this lets you find duplicate title/description/content issues and root out pages with broken links and invalid HTML you don't look at often.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#36

Post by wgjordan »

seanwilson wrote: post #34
wgjordan wrote: post #32

Earlier quoted context omitted.

Similar feature-set at first glance, but not open-source, not free after beta ends, and created/maintained by an unknown solo developer. Nice site design, but I think the cards are pretty heavily stacked against this one.

The big difference is Checkbot crawls whole websites as opposed to checking a single page at a time. The Checkbot interface is designed around helping you hunt down issues that impact groups of pages and pages you didn't think to check. For example, this lets you find duplicate title/description/content issues and root out pages with broken links and invalid HTML you don't look at often.

That does sound like a key differentiating feature, thanks for clarifying. While I'd probably prefer to hook up an open-source web-crawler to lighthouse (e.g., something like github/lightcrawler [1]), I could see SEO/marketing experts in particular paying for a user-friendly all-in-one solution like this versus cobbling something together from open-source tools.

[1] https://github.com/github/lightcrawler

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#37

Post by scrollaway »

seanwilson wrote: post #33
scrollaway wrote: post #31

Earlier quoted context omitted.

> Allowing Checkbot to login could help but I'll look into how to improve this. Wouldn't help in my scenario FWIW, my site is oauth-only login. > Hmm, any more suggestions on what to change here? I move the copy link into a clipboard button next to the url (like Github's "clipboard" button next to URLs), and make the remainder of the links prominent buttons. I would also avoid taking users to a separate page when cli…

> Wouldn't help in my scenario FWIW, my site is oauth-only login. Would being able to set cookies or sending custom headers help? > Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it. Hmm, so if you check "Explore" -> "External URLs" there's a ton of external links being ch…

> Would being able to set cookies or sending custom headers help?

I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding it myself.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#38

Post by seanwilson »

wgjordan wrote: post #36
seanwilson wrote: post #34

Earlier quoted context omitted.

The big difference is Checkbot crawls whole websites as opposed to checking a single page at a time. The Checkbot interface is designed around helping you hunt down issues that impact groups of pages and pages you didn't think to check. For example, this lets you find duplicate title/description/content issues and root out pages with broken links and invalid HTML you don't look at often.

That does sound like a key differentiating feature, thanks for clarifying. While I'd probably prefer to hook up an open-source web-crawler to lighthouse (e.g., something like github/lightcrawler [1]), I could see SEO/marketing experts in particular paying for a user-friendly all-in-one solution like this versus cobbling something together from open-source tools. [1] https://github.com/github/lightcrawler

Yes, so I think the UI is a really important factor here in terms of productivity and ease of use. For example, after Checkbot has scanned your localhost/development site and identified a page has the same title as other pages, you can edit your site, hit the "recrawl" button for that page and confirm your fix worked in a few seconds. Users I've worked with so far have really appreciated this fast and simple workflow.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#39

Post by seanwilson »

scrollaway wrote: post #37
seanwilson wrote: post #33

Earlier quoted context omitted.

> Wouldn't help in my scenario FWIW, my site is oauth-only login. Would being able to set cookies or sending custom headers help? > Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it. Hmm, so if you check "Explore" -> "External URLs" there's a ton of external links being ch…

> Would being able to set cookies or sending custom headers help? I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding…

> these pages won't get crawled by Google anyway.

Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this.

> I didn't have much success adding it myself.

The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum.

A common scenario I see as well is you start a crawl, see the URLs flying by and think "oops, don't want to crawl those URLs". A cancel button would help but hopefully more can be done.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#40

Post by seanwilson »

santoshmaharshi wrote: post #35

Very nicely thought through and implemented i agree with you on the best practice point. Already forwarded to few folks and going to stay with my browser.

Thanks! Let me know if you can think of any improvements I can make.
Post reply on HN