RFC 2616[0] says in section 14.43 that user agents SHOULD send a User-Agent field. Is there any particular reason why it is still sensible to send such a field at all, rather than maybe something more generic like ‘Crawler’, ‘Graphical’ and ‘Terminal’ (to distinguish the three cases that might come to mind immediately)? In other words, what benefit does a User-Agent field have nowadays when people are told to check f…
It is unlikely that you will want to block a traditional web browser, but web browsers are not the only clients available and some will be performing operations that are not necessarily in line with the wishes of your resources.