Robots.txt as a Security Measure?
cdsrc.com
Robots.txt as a Security Measure?
1–10 of 36 posts
Re: Robots.txt as a Security Measure?
#2There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
Re: Robots.txt as a Security Measure?
#3There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
It's far from a joke.
Re: Robots.txt as a Security Measure?
#4"I don't always expose my production database on a public URL, but when I do, I put a 'Disallow' in my robots.txt for it."
Re: Robots.txt as a Security Measure?
#5There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
A few years ago I purposefully put a couple of "interesting" paths in the robots.txt as a honeypot to test/capture bot conformance and malicious actors. Not one hit ever.
Re: Robots.txt as a Security Measure?
#6"I don't always expose my production database on a public URL, but when I do, I put a 'Disallow' in my robots.txt for it."
It reminds me of that scene from Spaceballs.
"The combination is... 1-2-3-4-5."
"That's amazing! I've got the same combination on my luggage!"
Re: Robots.txt as a Security Measure?
#7There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
In addition to the obvious that is literally a list of places where admins don't want to look, it is also often useful in backend technology enumeration.
Re: Robots.txt as a Security Measure?
#8There's a running joke among web pentesters about robots.txt being the first place you look when hitting a new site.
A while back I wrote a Python script to watch for links posted on Twitter and then scrape their /robots.txt file [1]. The requests are routed through Tor for privacy purposes.
It's been incredibly enlightening. One thing that sticks out immediately is that you can identify the underlying HTTP framework in many cases due to the defaults. Sometimes even the exact version.
And, yes, people do use the robots file to "protect" or "hide" endpoints and they can effectively be used to enumerate potential endpoints worth investigating further (from a pentesting perspective).
[1] https://gist.github.com/wybiral/20c20ccf00b6c93506b8acdc6ccb...
Re: Robots.txt as a Security Measure?
#9Honestly, I think it couldn't hurt, if done appropriately. If crawlers are indexing those pages, then they're publicly available anyway, and could be crawled by a determined attacker - so nothing in robots.txt ought to be truly sensitive. But if there's pages that ought to be secure, but might contain an exploitable vulnerability, putting their path in robots.txt at least limits their exposure to those determined enough to look, rather than any lazy script kiddie using Google to search your site.
Obviously you shouldn't rely on it, but defense in depth as always.
Re: Robots.txt as a Security Measure?
#10i move my not-to-be-indexed stuff around a lot: renaming, archiving, etc. & i've a bit of shell scripting and a commonlisp program that automatically add things to robots.txt so that barely any of the things listed won't 404, and the ones that won't are protected via htaccess.
not sure why i did this aside from that it was fun!