11–20 of 20 posts
This definitely solves a good problem. Company don’t keep generally good confluence docs and documentation. Somehow if there is common source of truth it helps to entire org in a company. But I was wandering if it will be helpful or not for external world because I feel companies usually double check any information before releasing it publicly ..especially related to code base.(Just a thought).
would be cool to extend this to enable auto-creating a pr to update a docs repo!
This definitely solves a good problem. Company don’t keep generally good confluence docs and documentation. Somehow if there is common source of truth it helps to entire org in a company. But I was wandering if it will be helpful or not for external world because I feel companies usually double check any information before releasing it publicly ..especially related to code base.(Just a thought).
> The snippet from "search docs crawling indexing pause online business" states that adding a Disallow: / rule for Googlebot in robots.txt will keep Googlebot away permanently as long as the rule remains. "search help office hours 2023 june", however, advises against disallowing all crawling via robots.txt, warning that such a file "may remove the website's content, and potentially its URLs, from Google Search." This directly contradicts the claim that a full-disallow rule safely blocks Googlebot without negative consequences, creating a true conflict about the effect and advisability of using a disallow rule to block Googlebot.
If you want to block Googlebot "permanently", why would you expect to stay listed in Search? The first page actually agrees with the second - if you only want to temporarily block crawling, it recommends not blocking Googlebot.
Actually, your last "conflict" is bad too. A 503 fetching robots.txt does stop crawling the site, for at least twelve hours and possibly forever (if other pages return errors). The only crawling Google will continue to do is to keep trying to fetch robots.txt.
I appreciate what you're trying to set up here but 2/4 is a pretty bad record for a demo.
Huh? The first "conflict" you list isn't a conflict. > The snippet from "search docs crawling indexing pause online business" states that adding a Disallow: / rule for Googlebot in robots.txt will keep Googlebot away permanently as long as the rule remains. "search help office hours 2023 june", however, advises against disallowing all crawling via robots.txt, warning that such a file "may remove the website's content…
I somewhat disagree with you on the last conflict: you have one document stating pretty clearly that returning a 503 for the robots.txt "blocks all crawling". The other document states there's a 12 hour block after which Google may decide to crawl the other pages (not just the robots.txt like you said).
Thanks for the feedback though, definitely some work to be done on validating the conflicts we surface.
Love the idea. Detecting when company truths differ between tools is a tricky technical problem, especially when the wording could differ between each tool.
Great concept! Worth trying :)