I'd rather have a site showing how well my site is protected from being accessed by AI agents would be preferable, and advises how I can lock it down further. Basically, the exact opposite of this.
[deleted]
Scan your website to see how ready it is for AI agents
181–190 of 190 posts
Re: Scan your website to see how ready it is for AI agents
#182Earlier quoted context omitted.
Maybe we can start a new protocol where the html is encrypted, and the viewer must try 2^10 to 2^20 hashes before the decryption key is discovered. Same formula that BTC mining uses. It would be negligible cost for any single user but terribly expensive for crawling en-masse.
Or we make a separate, smaller internet for humans with a verified identity.
> with a verified identity
Folks added an optional field to store a broad age category to optionally present to websites to facilitate keeping children out of porn sites, and everyone lost their collective minds.
Re: Scan your website to see how ready it is for AI agents
#183I tried this with both claude and gemini cli and it caused issues (that were easy to fix). Nice tool, but seems ai coded and needs refinement.
Re: Scan your website to see how ready it is for AI agents
#184AI industry: "AI agents will soon be able to do any white-collar human job!" Also AI industry: "Please make sure your website is adapted so that AI agents are able to use it."
Yes. But to be fair there are plenty of websites that are bad enough they’re not adapted for me to be able to use it.
Re: Scan your website to see how ready it is for AI agents
#185Ironically, this feels exactly like the various "semantic web" initiatives, only this time coming directly from the tech megacorps and not the starry-eyed "free web"/"open data" idealists. It will hit exactly the same walls too, namely that the technical details are completely irrelevant - if adopting a standard is actually a negative for websites, because it will separate the site from its users, sites will obviousl…
Semantic web all over again. The incentive problem is the same one that killed RDFa and microformats. Nobody maintains structured markup for someone else's benefit unless there's traffic or money at the end of it.
Re: Scan your website to see how ready it is for AI agents
#186We run agents on web tasks and the difference between
clean semantic HTML and a site drowning in React state is massive.
Agents can technically navigate both, but one costs 10x more in
compute and fails 5x more often.
Whether site owners have any incentive to care is a different question. Agent traffic doesn't pay the bills yet. Something like x402 could change that, but the payment rails aren't there.
Re: Scan your website to see how ready it is for AI agents
#187I'd rather have a site showing how well my site is protected from being accessed by AI agents would be preferable, and advises how I can lock it down further. Basically, the exact opposite of this.
I am building a product to help with this, please write me (email in profile), I would love to hear more about what you're trying to protect.
It's just a blog. Nothing fancy, nor important.
Re: Scan your website to see how ready it is for AI agents
#188Earlier quoted context omitted.
This still doesn't really answer my question, though. This is like telling me my old blog posts can't be parsed by your regex. Like... yeah, no shit; I didn't build it for your regex. It's not the target audience. Plus, isn't the appeal of LLMs broadly that they can do somewhat-useful things with mostly-arbitrary input (if you ignore the risk of prompt injection)?
> Plus, isn't the appeal of LLMs broadly that they can do somewhat-useful things with mostly-arbitrary input (if you ignore the risk of prompt injection)? They can definitely read HTML, but they do better with more structure. I proposed in a sibling comment for example that the "reader mode" feature in browsers might be a great LLM-compatibility feature to reduce all the HTML token noise. Or exposing an HTTP API with…
I don't think it's fair to say that HTML's less structured than Markdown. Markdown is derived from a simplified subset of HTML, and having myself cut my teeth on HTML5 when it was still new, there's been a huge emphasis on the idea of the semantic web conveyed through HTML.
Re: Scan your website to see how ready it is for AI agents
#189[dead]
Re: Scan your website to see how ready it is for AI agents
#190[flagged]