421–430 of 454 posts
Earlier quoted context omitted.
What happens when someone is from a shit country?
AFAIK you already have to use VPN if you are e.g. westerner visiting China or Belarus.
It feels like you are a third class citizen.
> This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
Thank you, from Vietnam.
An interesting article, and an interesting problem to have! The problem, to recap the title, is that "99% of My Website Traffic Is Bots". How to fight all of those bots, all of those web-scrapers... If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online ca…
Earlier quoted context omitted.
FYI, your proof is hosted on a NSFW page. You should have mentioned that.
Reason #13843 for using an ad blocker.
Can someone help me understand the underlying motivation behind this? It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely. What's the…
Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it. It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.
Earlier quoted context omitted.
How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices? Not a rhetorical question, because most companies would welcome that interest from customers.
> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices? That's the thing; they can't, really. Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of mic…
Earlier quoted context omitted.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.
you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…
I'm not sure who's the corpo here. Most likely not OP.
Earlier quoted context omitted.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
Per personal communication, this is precisely why the developer did that.