The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor for the rest of us.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
