Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Checkbot for Chrome – web crawler that tests for web best practices

checkbot.io

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#11

Post by seanwilson »

ajeet_dhaliwal wrote: post #7
seanwilson wrote: post #6

Earlier quoted context omitted.

The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development b…

Who’s the customer you have in mind who will want to do this for thousands of pages?

People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#12

Post by seanwilson »

cdawg0 wrote: post #10
seanwilson wrote: post #6

Earlier quoted context omitted.

The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development b…

Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?

> Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process?

A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well.

> Also, does Checkbot dig into all dependancies or skip them like some others?

Can you expand on what you mean here?

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#14

Post by seanwilson »

Raphmedia wrote: post #13

Works pretty good. Incidentally, I just locked myself from my own server. This double as a security tester! :o)

On the left sidebar at the start you can modify the number of URLs crawled per second if that helps!

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#16

Post by idle_processor »

seanwilson wrote: post #2

Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and security best practices so I'd love to know what the community thinks. Can you think of any changes that would make Checkbot more helpful? Did you notice any bugs? Thanks!

- A regex blacklist for URL structure would be very helpful.

- Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...).

Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip).

(E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)

Re: Show HN: Checkbot for Chrome – web crawler that tests for web best practices

#20

Post by chasers »

seanwilson wrote: post #18
chasers wrote: post #17

It's nice and fast! Are you actually using chrome to render each page or making requests some other way?

Thanks! All requests are done from your own browser if that's what you mean.

No, I mean like render the whole page with JS and all, but from your FAQ it seems like you're not. Which is why it's fast, ha.
Post reply on HN