Live data from Hacker News

I rebuilt my blog's cache. Bots are the audience now

hoeijmakers.net

21–30 of 64 posts

Re: I rebuilt my blog's cache. Bots are the audience now

#21

Earlier quoted context omitted.

I pulled all the websites I had - some existed for a decade plus and made me hundreds of thousands of dollars. All that is left is bots that theft the value of my work. Until something changes, goodbye.

This is like choosing to be an elementary school teacher and then quitting because it turns out your students for the year aren't your pets in perpetuity.

If your students were growing up to subvert your line of work, sure. Pretty sure that’s not the case though!

Re: I rebuilt my blog's cache. Bots are the audience now

#22
post #2

Personally, I think this is a good idea. But the core problem is this: How is a newcomer supposed to build reputation now? Without exaggerated business promises or capital, basic online reputation usually depends on writing. In fact, my own first step into freelancing came because someone found the articles on my Korean blog interesting. So the question is: if the subscribers are bots, what benefit do they actually g…

I feel that pressure of not knowing how to definitively compete on the internet, especially when there's so much AI created noise.

I'm a copywriter and I used to get hired to write posts on behalf of founders on LinkedIn or for their company blog.

Now, the last three jobs I had were all focused on sending cold email.

Re: I rebuilt my blog's cache. Bots are the audience now

#23
I went through a similar process recently. For a while I saw readership of my site gradually increasing, and eventually it became clear that it wasn't human beings.

I also used Claude to help me drill into what's going on. Bizarrely, about 80% of my traffic comes from Singapore, which the author mentioned. I don't know why. A lot of the traffic looks real; it stays for a while, clicks different links in different orders. But no one in Singapore has ever read a thing I've written on my site as far as I'm concerned.

I thought Cloudflare would help protect my site from bots, but it utterly fails. I'm not sure if they're too sophisticated or people overestimate how well CF works for these things. I paid for advanced features for a while and reverted to the free plan once I realized it made no difference. It's a great platform in general, but hasn't been great for allowing me to see how many humans actually read my content.

I know some do because they email me occasionally. If I had to guess, of the ~200 visits per week reported in analytics, around 15 are real.

Re: I rebuilt my blog's cache. Bots are the audience now

#24
I’m very confused about why you’d have such a complex cache arrangement. Sounds like you’re using Cloudflare and Fastly to do roughly the same thing. That sounds like a recipe for more expense and more problems.

For the sort of thing you’re doing, it should be as simple as “throw it behind Cloudflare/Fastly/Bunny/whichever private CDN you like” and that’s it.

Also the diagram near the end is pretty much incoherent. GenAI, I presume.

Re: I rebuilt my blog's cache. Bots are the audience now

#27
post #16

> Not because I expect a person in Singapore to shave 200ms off their pageload, but because the next request for that page is more likely to come from a retrieval system than a browser, and the request after that, and the one after that. Why do I care if I shave off 200ms from a crawler's request, instead of a human's?

From the post: > If you care about how your content moves through the world now, including through AI systems, you have to care about caching. Not as a performance optimisation for human browsers, but as infrastructure for machine readership.

That doesn’t answer the question at all, and I wonder if it’s actually true? A cache is not magic; it is, itself, just a static file server in the end. If I self host a static page website on an nginx box, do I actually need cache to serve today’s crawlers?

The screenshot in the image says 3k req/day. That’s 2 requests per minute (amortized). At that rate, you can serve it with cgi and Perl.

Cache is only relevant if you have a lot of traffic AND dynamic pages, or if you care about latency (which is only relevant for humans).

Re: I rebuilt my blog's cache. Bots are the audience now

#28
post #2

Personally, I think this is a good idea. But the core problem is this: How is a newcomer supposed to build reputation now? Without exaggerated business promises or capital, basic online reputation usually depends on writing. In fact, my own first step into freelancing came because someone found the articles on my Korean blog interesting. So the question is: if the subscribers are bots, what benefit do they actually g…

I think in general, "Writing on the Internet with the intent to make money" is effectively dead, or at least soon to be dead. AI+bots mean we now have the "infinite typewriter monkeys" from the thought experiment. With infinite supply, the price goes to zero.

We need to stop this treadmill of trying to "build reputation" and stop focusing on "symbolic capital" and "clout" and whatever else bloggers are going after. You're not going to get it, and even if you do, you're not going to be able to "monetize" it.

If you have a need to write, write. Maybe a handful of actual people will read it, maybe not. But, I wouldn't try to do it for a living. The reward will have to be the cathartic process of writing itself, and not in how much attention it gets, how much it "blows up" or how viral it gets.

Re: I rebuilt my blog's cache. Bots are the audience now

#29
my tiny blogs no one reads have been racking up a huge deno deploy and vercel bills ($40-50/mo each) bc I ran them "naked" without a cache or cloudflare or static builds - it didnt matter bc I got like hundreds of visitors a month. they were just hono or whatever api pulling from my backend which could be notion or airtable - super simple, though kind of slow

now I suddenly I have 10k visitors a month hammering my apis and causing massive egress and cpu usage - so i had to get them behind cloudflare and now build everything statically - cut the costs back down from 90+ cpu hours to about 0.2 cpu hours a month

crazy times

(also, all donw w/ claude code's help, or it would have taken a week for me to figure out)

Re: I rebuilt my blog's cache. Bots are the audience now

#30
Yesterday I logged into cloudflare and found that Cloudflare had blocked chatgpt and claude from accessing my site. https://macrocodex.app

This is bad because there are fitness guides on my domain

https://macrocodex.app/guides which newbies often put in chatgpt and asks to simplify.

I enabled crawl for LLMs. There is lot of misinformation in fitness field so it's better if LLMs get their content from people who atleast have experience in the field

Post reply on HN