Live data from Hacker News

Show HN: Sitebulb, a website crawler and auditor for SEOs

sitebulb.com

51–60 of 75 posts

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#51

We're nobody special. We're not a cool startup that's just secured funding, we're a bootstrapped, 2 man team and we've built both our products from the ground up. Their honesty bought me! Nice going bros.

:) thanks

You might like these as well: https://sitebulb.com/release-notes/

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#52

What's this doing differently or better than screamingfrog? Which is also a desktop program and provides quite robust information. SF has been one of the standard industry tools for those doing SEO for years.

SF is also a lot cheaper. With VAT, you're looking at £705.60 for sitebulb per year, vs £149.00 per year for SF. That's a really big price difference and you'd have to be really sure it's worth it. In addition SF works on Ubuntu, which is another point in its favour.

I know right, SF is just too cheap for its own good! :p

We think it's a case of horses for courses. Sitebulb has the potential to save you a ton of time when auditing and reporting. If you don't do a lot of that then it might not be a good option for you. If you do, that's where a lot of the value lies.

There's a fully featured 2 week trial to give it a proper go, and the monthly billing means you have the option to switch it on/off as you need it.

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#54
post #50

Hey HN, Sitebulb is a desktop crawler for Windows and Mac, specifically designed for SEO consultants/agencies. Me and my business partner have been building it for the last 2 years or so, and we're finally launching it today. It's main differentiating factors: 1. Scale – it can comfortably crawl 500,000+ page websites despite being a desktop program. 2. Reporting – it does a lot of data manipulation and processing so…

Some feedback: - Why require the email confirmation before using the software? Not really necessary, is it? - No Umlauts in project name? - Standard/advanced settings switcher is confusing - Crawl Maps is not linked from the "Product" dropdown - "Recent audits" shows finished and queued ones, but not running ones (which also have no menu point) - Super simple option to limit crawl to "internal" URLs would be nice (or…

Thanks for all the detail! Here you go:

- Email confirmation is required for the username/password, which is how free and trial licenses are controlled, and ultimately how paid licenses are doled out. So we need it for the licensing.

- No special characters at all! Excepts periods. Sorry!

- Agreed, we need to improve the settings switcher.

- Crawl Maps is not linked - you mean on the website right? I'll fix that.

- Running audits show on the main Dashboard, seemed kinda overkill to put it on Recent Audits as well. No?

- You can switch of 'Check external' in the Advanced Settings. Kinda 'hidden away' to keep the main settings UI cleaner (otherwise where does it end?!)

- "Filtered URL Lists" - they are there because people want them ('a big list of all the URLs') and kept missing them in our usability tests!

- Why no endless scrolling in tables? It's not easy to do because the data is written to disk, rather than stored in RAM (which is the reason it can typically crawl more pages), so it needs to go and fetch/filter/etc... every time.

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#56

Hey HN, Sitebulb is a desktop crawler for Windows and Mac, specifically designed for SEO consultants/agencies. Me and my business partner have been building it for the last 2 years or so, and we're finally launching it today. It's main differentiating factors: 1. Scale – it can comfortably crawl 500,000+ page websites despite being a desktop program. 2. Reporting – it does a lot of data manipulation and processing so…

The problem is for big crawls( and 500k is not large) you probably don't want to use your desktop for example my home adsl is only 3.5 as we are 6kyards from the exchange.

And I would not want to get my works 100Mb banned by google. This is where services like deep crawl come in to play I can set up my sites to be crawled at night and look at the reports in the morning.

And another problem I found is desktop crawlers are very resource hungry at one small agency we had two striped down dedicated machines just to run crawls as the risk of causing a crash was to high

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#58

What's this doing differently or better than screamingfrog? Which is also a desktop program and provides quite robust information. SF has been one of the standard industry tools for those doing SEO for years.

The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc... I've written a more comprehensive answer to this here: https://sitebulb.com/resources/guides/how-is-sitebulb-differ...

I'm testing Sitebulb right now (trial version). The crawling is kinda slow (I'm on 100mbits fiber). Why did you choose to build eveything from scratch instead of making an application that use the results from other crawlers/spiders (ie: Screaming Frog) and just produce the audit reports?

EDIT:

And after about 3hours of crawl, this is what I got (and no way to resume it):

>Audit Stopped! >The audit stopped early because: Maximum Crawl depth limit of 50 reached

>WARNING: Audit Paused ! The audit is incomplete and did not finish properly.

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#59
post #58

Earlier quoted context omitted.

The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc... I've written a more comprehensive answer to this here: https://sitebulb.com/resources/guides/how-is-sitebulb-differ...

I'm testing Sitebulb right now (trial version). The crawling is kinda slow (I'm on 100mbits fiber). Why did you choose to build eveything from scratch instead of making an application that use the results from other crawlers/spiders (ie: Screaming Frog) and just produce the audit reports? EDIT: And after about 3hours of crawl, this is what I got (and no way to resume it): >Audit Stopped! >The audit stopped early beca…

I left the program running in the background and it resumed itself after a few minutes. I have no idea why as there was no info on the dashboard.

Re: Show HN: Sitebulb, a website crawler and auditor for SEOs

#60
Hmm, I set it to crawl our little Ghost blog (e.g. blog.example.com) and it immediately jumped to spidering our main site (e.g. www.example.com). Now, I imagine if I had properly poked at the Advanced settings I could have limited it to the initial subdomain, but I would have expected that to be the default...
Post reply on HN