Edit: thanks for those links! I'm somewhat out of the loop academically, so have been relying on search engines whose quality seems to be in decline.
-
Combatting bots with the Cloudflare challenge screen is an X/Y problem.
The central issue is that the web has been rolled out improperly, and the way that we build websites is incorrect. The web should have been decentralized, meaning that all public-facing pages would be public domain and hosted on a peer to peer (P2P) network that grows more powerful with the number of users, similarly to how BitTorrent works. We wouldn't concern ourselves with servers at the edge, since they would already be distributed around the world and implement the caching strategies that are already part of HTTP.
Which means for example that regions in AWS would be unnecessary, and Cloudflare and other content distribution networks (CDNs) would have no business model. Coral CDN was a free working example of automatic caching that ran up until a few years ago:
https://wiki.opensourceecology.org/wiki/Coral_CDN
https://en.wikipedia.org/wiki/Coral_Content_Distribution_Net...
https://cachedview.com
https://news.ycombinator.com/item?id=19020978
Note how it's mostly been erased from history due to ensh@ttification by FAANG.
It also means that web technologies we think of as core to how external resources are included are also incorrect. Rather than Cross-Origin Resource Sharing (CORS), we should be using Subresource Integrity (SRI). That would allow us to include scripts and other media files by hash instead of just location. That also removes most of the need for build processes like Webpack, Grunt, Gulp, etc, since scripts would import other scripts directly and let the Just in Time (JIT) compiler decide what is needed.
I can go on pretty much forever with this. In 1995 I was a student at the University of Illinois in Urbana-Champaign (UIUC) where NCSA Mosaic was developed, which Netscape copied the year before when it took the internet mainstream. Stuff like Server-Side Includes (SSI) showed promise in avoiding build tools by letting developers reuse code from other servers. But there wasn't full understanding then of how hashing makes strong security guarantees. In the meantime, Marc Andreessen and other billionaires took the quick and easy path, rolling out easier (but not simpler) technologies that maximize short-term profits instead of long-term prosperity and ease of maintenance through automation.
Without a true distributed web, the endgame of all this looks like what we're seeing today. Sites that can't be scraped by alternative search engines or machine learning tools. Sites that can't be viewed securely or anonymously with Tor Browser. Sites that keep everything behind a paywall or in walled gardens, which will cause most of today's human-produced media to eventually be lost to the digital dark age.
Fixing all of this is straightforward, but it would probably require us to return to traditional values. Basically contributing some of our incomes to universities and other institutions via our taxes, so that they can work to protect the interests of the masses, who have no benefactor because it's not profitable to help them.
Billionaires and other moneyed interests don't want this, so have done everything in their power to dismantle the commons, not just on the web, but through regulatory capture to sell off public lands and other resources currently owned by everyone:
https://www.snopes.com/fact-check/elon-musk-stop-donating-wi...
Which means that this is really a cultural issue, so many of us can't see the problems or solutions without challenging our most closely-held beliefs, which creates cognitive dissonance. So even though the fixes appear obvious, they are effectively out of reach for the foreseeable future because it's easier to sabotage the system than reform it.
None of this helps you immediately though. You might be able to move from Cloudflare to a free and open source alternative like CloudFIRE, although it looks like they are copying many of its same mistakes, for example "fake browser detection and blocking" which is at the top of their list of priorities:
https://github.com/coinkite/cloudfire
I'm having trouble finding other alternatives:
https://news.ycombinator.com/item?id=34800182
So this is what I mean. If you are really interested in empowering large groups of people with free access to information, then you will be running up against the full might and momentum of the status quo.
Something that gives me hope is that most hackers and makers were originally drawn to tech as a lifeline out of subjugation doing mundane and pointless work. Tech is inherently antiauthoritarian. So all it would take is a single wealthy individual, a single internet lottery winner, to fund efforts to reevaluate what underpins the status quo from first principles. It might not take much to deliver tech which can't be unseen, which routes around artificial scarcity. We can imagine providing resources through automation, outside of any profit motive. Until then, large groups of individuals will have to keep contributing to these efforts on their own dime at a snail's pace, with what little motivation they have left after working their lives away to make rent and enrich the already wealthy.
Apologies for the wall of text, but it's the holidays so why not.
