Why do I get just an empty page?
Thanks. It seems to be very local/incidental. The page works from the locations I can test, but I’ll check whether one edge cache or request path served a bad response.
White page from one of Germany's largest ISPs.
I rebuilt my blog's cache. Bots are the audience now
41–50 of 64 posts
Re: I rebuilt my blog's cache. Bots are the audience now
#42Why would I want bots to read my blog?
Like it or not, but it's the intermediary layer now, between you and your users.
Re: I rebuilt my blog's cache. Bots are the audience now
#43Earlier quoted context omitted.
What's worse, is they train on your content, and very often you don't even get an attribution link. So the end user never even knows it was your site that provided the information and you never even get a single clickthrough. It's not like the SERPs where someone would click through, read your site, hopefully find it interesting and useful and come back. It's going to be a serious problem and I've already seen sites…
I pulled all the websites I had - some existed for a decade plus and made me hundreds of thousands of dollars. All that is left is bots that theft the value of my work. Until something changes, goodbye.
I have no problem with these AI summaries but someone needs to make a law requiring the AI provider to generate links to the primary source as part of the output. If a human writer ripped off someones work and didn't attribute, it would be a huge scandal. when an AI does it, its expected.
Re: I rebuilt my blog's cache. Bots are the audience now
#44Personally, I think this is a good idea. But the core problem is this: How is a newcomer supposed to build reputation now? Without exaggerated business promises or capital, basic online reputation usually depends on writing. In fact, my own first step into freelancing came because someone found the articles on my Korean blog interesting. So the question is: if the subscribers are bots, what benefit do they actually g…
I dunno. SEO spam made by human-robots already killed the first half of the internet in the late 2010s. Mostly driven by low-paid freelancers writing without expertise and plagiarizing other articles like crazy.
It has been virtually impossible to find real information in some areas if you don't personally know which website is reliable or not. That's why we devs used to go to StackOverflow, and people use site:reddit.com when searching Google. LLMs just exacerbated all of that, but it was already happening.
Re: I rebuilt my blog's cache. Bots are the audience now
#45Why would I want bots to read my blog?
I recently un-denylisted AI crawler UAs from my blog's robots.txt, for several reasons:
- My blog is static content and it costs me ~nothing to serve the requests.
- The bots were ignoring robots.txt anyway.
- If there's ultimately a human driving the bot (e.g. someone asking "summarise this article"), I don't mind.
- It's like trying to block search engines. Just as I want my blog to turn up in search results, I want agents etc. to know it exists, too.
My original motivation for denylisting, years ago, was that LLMs were simply not very good, so training-set scrapers seemed like all downside with no upside.
Re: I rebuilt my blog's cache. Bots are the audience now
#46I went through a similar process recently. For a while I saw readership of my site gradually increasing, and eventually it became clear that it wasn't human beings. I also used Claude to help me drill into what's going on. Bizarrely, about 80% of my traffic comes from Singapore, which the author mentioned. I don't know why. A lot of the traffic looks real; it stays for a while, clicks different links in different ord…
Same ratio roughly. 80% Crawlers and agents, 20% human. Loads of the agents actually serve the content to humans, mostly in ChatGPT.
Wow, that's a great point... I hadn't considered that. I assumed it's all training.
From what I understand, Cloudflare is trying to create a way for agents to consume content in a more structured manner than allowed for attention to the author, and potentially payment along with it.
I don't want to be paid but I'd love to see how often context from my writing winds up in a session a human is actively using.
Re: I rebuilt my blog's cache. Bots are the audience now
#47Earlier quoted context omitted.
What's worse, is they train on your content, and very often you don't even get an attribution link. So the end user never even knows it was your site that provided the information and you never even get a single clickthrough. It's not like the SERPs where someone would click through, read your site, hopefully find it interesting and useful and come back. It's going to be a serious problem and I've already seen sites…
I pulled all the websites I had - some existed for a decade plus and made me hundreds of thousands of dollars. All that is left is bots that theft the value of my work. Until something changes, goodbye.
You might as well serve different content to bots. Incorrect content.
Re: I rebuilt my blog's cache. Bots are the audience now
#48I denylisted traffic from Singapore. As far as I could discern it was all Windows 11 machines running Chrome accessing pages sequentially. I’d love to not do that, but trying to sift through that is quite the task.
TLS fingerprinting may identify what they actually are
Re: I rebuilt my blog's cache. Bots are the audience now
#49Yesterday I logged into cloudflare and found that Cloudflare had blocked chatgpt and claude from accessing my site. https://macrocodex.app This is bad because there are fitness guides on my domain https://macrocodex.app/guides which newbies often put in chatgpt and asks to simplify. I enabled crawl for LLMs. There is lot of misinformation in fitness field so it's better if LLMs get their content from people who atlea…
It is good to make a proper distinction, in the ChatGPT context, between crawlers and agents. The crawlers go for the content to build a new model, the agents serve content to users. The last one can be very useful.
They use different user-agent strings. The crawlers obfuscate themselves and use residential proxies. The agents call themselves ChatGPT-User. Of course Cloudflare wants OpenAI to pay them for not blocking ChatGPT-User by default.
Re: I rebuilt my blog's cache. Bots are the audience now
#50I simply block all AI crawlers with a user-agent check in nginx.conf.
This works for a few weeks to months. Then they detect your site is hostile to them and enable evasion mode, with random IP addresses and user-agent strings. Proxies are expensive so at least they're losing money.