Live data from Hacker News

Show HN: Checkbot for Chrome – web crawler that tests for web best practices

checkbot.io

21–30 of 47 posts

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#21
Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems.

First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well.

I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice. I think my hatred of that term stems from seeing it plastered all over older MS docs and the usual crop of "me too" copy n paste blog postings that litter the web, not to mention various forum postings. We're all bloody experts who know best in this game 8)

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#22
Really cool stuff. Here's some initial feedback:

"Avoid internal link redirects" -> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects.

"Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc -> Same problem as above with login walls ("Sign in to ..."). I get why, but it's adding a ton of noise. I added /account/login to URLs to ignore but it didn't achieve anything, I'm guessing I must have misunderstood the syntax or it has to be an exact match or some such?

"Avoid inline JavaScript" -> I'm making quite a bit of use of the pattern, which allows me to declare json objects in the body I can later parse in my scripts. This pattern doesn't have all the issues carried with inline js. Can you ignore script tags where the type= is unknown or application/json\?

hsts preload: This is picking up individual pages on the checked site as errors, even though hsts preload really is a domain-wide thing.

"Hide server version data" -> This is picking up "server: cloudflare" as an error. Means no site behind cloudflare will ever pass this which seems overkill.

"Use lowercase URLs" -> So, on my site you can access objects with IDs like youtube's (/id/vZEz7JoNnfgVo...). It's picking up all those as errors. Feels wrong?

UI: Not a fan of the "x inlinks / y outlinks / headers / recrawl / html / copy" links below the URLs on the results page. Low contrast, unclear what I'm clicking and where it's gonna take me. The "copy" button: What am I copying? (Clearly the URL as I tried it, but that'd be more useful as a clipboard button next to the URL for example)

Finally, I ran it on my company's blog and it ended up crawling a ton of the company's various exosites on different domains which wasn't super useful, especially since none of it showed up in the final results.

Hey, this is a really great tool. Fast, slick UI and very clear what it does. I'll keep an eye on it and would love to see what else it can do in the future.

Pricing: It's hard to see myself pay for this; not because it's not worth it (I think it is easily worth a dozen USD / site checked), but because it's so easy to look at what it does and think "Yeah, but, I can probably do all that myself, and if I don't, it's not so important that I need to pay for a tool to tell me what to fix". I think this is the curse of developing products targeted at developers: Devs will tend to think "I can do this myself // I don't need this". In fact, if you hook me up with a free account, I'll use it a bunch ;)

Shoot me an email (see profile) if you want to talk through some more feedback (especially UX feedback). You just provided me with a pretty cool service for free so I feel I have to give back :)

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#23
post #4

Earlier quoted context omitted.

Cool, I'm checking it out right now. Can you share what your tool provides that others don't?

The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development b…

> The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss.

ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#25
post #21

Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems. First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its…

I agree, it runs nicely, and is nicely presented. The term best practice does irk me. But weirdly enough I think the use of that term is why I decided to give it a go.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#27
post #15

Nice. very similar to Lighthouse https://developers.google.com/web/tools/lighthouse/#devtools

Which is also available in Chrome DevTools via the Audits panel.

I've never been able to use that, every time I run the test it says initial page load is 10s, but I'm running it against a static website hosted locally.. how is it taking so long? Even if I visited my site from the other side of the world over 3g it would be faster than lighthouse accessing it locally.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#28
post #21

Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems. First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its…

> First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well.

Awesome, thanks for trying it!

> I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice.

Yes, I'll admit I don't love the phrase myself. I don't like when someone is pushing what's really just their opinion as the one and only way to do things but the phrase does get the point across when you've got a character limit.

I've been really careful with the rules in Checkbot so far and the guide (https://www.checkbot.io/guide/) has links to where web experts like Google, Mozilla, OWASP and W3C recommend each practice. For example, making sure every page has a title, HTTPS is enabled and compression is used is difficult to argue against. Let me know if you think any of the rules need some changes however.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#29

Really cool stuff. Here's some initial feedback: "Avoid internal link redirects" -> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc -> Same problem as above with login walls ("Sign in to ..."). I get…

Awesome, thanks for the detailed feedback! As you can probably imagine, tweaking the rules to work with every imaginable website configuration is an ongoing process so this is super helpful.

> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects.

> "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc

Allowing Checkbot to login could help but I'll look into how to improve this.

> "Avoid inline JavaScript" -> I'm making quite a bit of use of the

Ah, thanks, this is an easy fix. I'm planning to add structured data checks in the future as well because checking you've configured these correctly on all your pages is cumbersome.

> "Hide server version data" -> This is picking up "server: cloudflare" as an error. Means no site behind cloudflare will ever pass this which seems overkill.

Yes, for what it's worth this is defined as "low priority" internally and the rule description is written to emphasise this. I could change it to only fire when there's version numbers in the headers perhaps. I agree knowing you're using Cloudflare isn't a big deal but some servers will advertise very specific OS and PHP versions for example.

> "Use lowercase URLs" -> So, on my site you can access objects with IDs like youtube's (/id/vZEz7JoNnfgVo...). It's picking up all those as errors. Feels wrong?

Yes, I'll need to think how to avoid that case. It's a good general rule when you're writing human readable URLs however so I wouldn't want to disable it completely.

> UI: Not a fan of the "x inlinks / y outlinks / headers / recrawl / html / copy" links below the URLs on the results page. Low contrast, unclear what I'm clicking and where it's gonna take me. The "copy" button: What am I copying? (Clearly the URL as I tried it, but that'd be more useful as a clipboard button next to the URL for example)

Hmm, any more suggestions on what to change here? I made these links prominent because they were common user actions and added tooltips to them to help describe what they do. I agree it's not completely obvious what they do at first but there's only so much space available. I experimented with only showing these when you hover over a table cell. I do want to add more shortcuts in the future such as a quick way to look up a URL on Google or on archive.org so I'll likely have a "more" button for extra options later.

> Finally, I ran it on my company's blog and it ended up crawling a ton of the company's various exosites on different domains which wasn't super useful, especially since none of it showed up in the final results.

Can you give more details here? Checkbot will probe links to check they're working for example but shouldn't spider sites that are considered external. I originally had it crawling subdomains of the start URL but changed that default because it wasn't what most people wanted.

> Shoot me an email (see profile) if you want to talk through some more feedback (especially UX feedback).

Great, let's keep in contact (see my profile as well)! Hopefully it's obvious UX is important to me too. It's been challenging to find a balance in showing the right amount of information on the screen while battling with the horizontal space constraints you get with long URLs. The "Avoid temporary redirects" report is a good example of this e.g. for each row, you want to know the redirect status code, the start URL, redirect destination and redirect path.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#30

Earlier quoted context omitted.

Which is also available in Chrome DevTools via the Audits panel.

I've never been able to use that, every time I run the test it says initial page load is 10s, but I'm running it against a static website hosted locally.. how is it taking so long? Even if I visited my site from the other side of the world over 3g it would be faster than lighthouse accessing it locally.

Part of the audit includes throttling your connection to simulate a device on a slower link.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/...

Post reply on HN