On a similar note, tools like [grabsite](https://github.com/ArchiveTeam/grab-site) wisely use robots.txt as a method of finding additional paths to archive when crawling sites.
Robots.txt as a Security Measure?
21–30 of 36 posts
Re: Robots.txt as a Security Measure?
#22I was hoping this would be about putting an orphan path in your robots.txt and then black-listing clients who tried to fetch it -- nobody should know about it except robots who are told not to go there, so anyone who visits the link is an adversary.
Ooo that's a good idea, I will definitely start implementing this
Re: Robots.txt as a Security Measure?
#23I was hoping this would be about putting an orphan path in your robots.txt and then black-listing clients who tried to fetch it -- nobody should know about it except robots who are told not to go there, so anyone who visits the link is an adversary.
Funny experiment and perhaps also useful, but there are crawlers with good intentions[1] that still may ignore the disallows. I don't know of anyone else than the internet archive though.
[1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Re: Robots.txt as a Security Measure?
#24I was hoping this would be about putting an orphan path in your robots.txt and then black-listing clients who tried to fetch it -- nobody should know about it except robots who are told not to go there, so anyone who visits the link is an adversary.
Definitely an interesting idea, I should check the scene to figure out how many 'adversaries' are actually scanning robots.txt files.
Re: Robots.txt as a Security Measure?
#25Just put a fake /wp/admin/login URL (or similar) in the disallow rules, then just IP ban everyone trying to access it for 24 hours. That's how you do robots.txt Security.
Re: Robots.txt as a Security Measure?
#26I was hoping this would be about putting an orphan path in your robots.txt and then black-listing clients who tried to fetch it -- nobody should know about it except robots who are told not to go there, so anyone who visits the link is an adversary.
Funny experiment and perhaps also useful, but there are crawlers with good intentions[1] that still may ignore the disallows. I don't know of anyone else than the internet archive though. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Those crawlers can almost always be recognized by the UA.
Re: Robots.txt as a Security Measure?
#27Why in the world would you put things that shouldn't be downloaded ON THE INTERNET to begin with.. If you then proceed to also tell the whole wide world that you did it.. It's difficult to feel any empathy.
Re: Robots.txt as a Security Measure?
#28Earlier quoted context omitted.
Funny experiment and perhaps also useful, but there are crawlers with good intentions[1] that still may ignore the disallows. I don't know of anyone else than the internet archive though. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Those crawlers can almost always be recognized by the UA.
Yes, no doubt.
Re: Robots.txt as a Security Measure?
#29Earlier quoted context omitted.
Meanwhile over in .gov I’ve had to explain to a pentester that it wasn’t a security problem that robots.txt was accessible without authentication, based on a very big vendor’s scanner having badly regurgitated the OWASP advice.
The "security" world has an unusually high level of total incompetence. It is scary.
This is common any time there’s so much demand: in the late 90s it was not uncommon to be in a room full of people who were ostensibly web developers and didn’t understand how the web or their backend servers worked but were certain they were about to become rich.
Security is especially bad because so many large organizations are under pressure to improve but the market is tight and the pool of experts is limited. Also, many places have outsourced to large contracting companies who don’t want to admit they don’t have enough qualified staff and will hope that you’ll be satisfied with whoever they deliver.
Re: Robots.txt as a Security Measure?
#30Earlier quoted context omitted.
The "security" world has an unusually high level of total incompetence. It is scary.
This is common any time there’s so much demand: in the late 90s it was not uncommon to be in a room full of people who were ostensibly web developers and didn’t understand how the web or their backend servers worked but were certain they were about to become rich. Security is especially bad because so many large organizations are under pressure to improve but the market is tight and the pool of experts is limited. Al…
Yeah no doubt it is a phase.
It's just a really nasty phrase right now.
I always think of this:
https://medium.com/@djhoulihan/no-panera-bread-doesnt-take-s...