11–20 of 36 posts
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
This is the only way to use robots.txt for semi-sensitive info, and obviously not for info so sensitive that it would be awful for it to get out. URLs can leak through proxy logs and shared browser history.
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
A few years ago I purposefully put a couple of "interesting" paths in the robots.txt as a honeypot to test/capture bot conformance and malicious actors. Not one hit ever.
Honestly, I think it couldn't hurt, if done appropriately . If crawlers are indexing those pages, then they're publicly available anyway, and could be crawled by a determined attacker - so nothing in robots.txt ought to be truly sensitive. But if there's pages that ought to be secure, but might contain an exploitable vulnerability, putting their path in robots.txt at least limits their exposure to those determined en…
https://developers.google.com/search/reference/robots_meta_t...
I’d also strongly recommend pairing this with outside monitoring which alerts if something accidentally becomes reachable since it’s really easy not to notice something working from more places than intended.
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
Honestly, I think it couldn't hurt, if done appropriately . If crawlers are indexing those pages, then they're publicly available anyway, and could be crawled by a determined attacker - so nothing in robots.txt ought to be truly sensitive. But if there's pages that ought to be secure, but might contain an exploitable vulnerability, putting their path in robots.txt at least limits their exposure to those determined en…
There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
Meanwhile over in .gov I’ve had to explain to a pentester that it wasn’t a security problem that robots.txt was accessible without authentication, based on a very big vendor’s scanner having badly regurgitated the OWASP advice.