[deleted]
Serve Markdown to AI Agents with Accept Headers
41–50 of 105 posts
Re: Serve Markdown to AI Agents with Accept Headers
#42No, I don't think I will. I publish things for people, not bots.
Re: Serve Markdown to AI Agents with Accept Headers
#43Best case scenario, this ends up being abused in order to feed LLMs crap responses (or worse).
Re: Serve Markdown to AI Agents with Accept Headers
#44I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…
>On the caching impact, Simon Willison wrote: Wrong: https://developers.cloudflare.com/cache/concepts/vary/
Yes. This feature is brand new: https://developers.cloudflare.com/cache/changelog/
Simon wrote that in 2023: https://simonwillison.net/2023/Nov/20/cloudflare-does-not-co...
Re: Serve Markdown to AI Agents with Accept Headers
#45Is this an alternative or complements llms.txt?
Re: Serve Markdown to AI Agents with Accept Headers
#46I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption. I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
I mean, it's exceptionally easy for them to add it. Even with low adoption. Somewhere in the bowels of chatgpt there is code that queries for html, and then parses that to markdown or something similar. All this would be is adding the logic to request markdown if available, then bypass the html->markdown step if you get markdown back.
Re: Serve Markdown to AI Agents with Accept Headers
#47Is this an alternative or complements llms.txt?
[dead]
Re: Serve Markdown to AI Agents with Accept Headers
#48What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
I agree. I think a lot of people here are assuming that the full HTML retrieved has to go into the LLM eating up tokens. But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM. Semantic HTML would make that easier.
Re: Serve Markdown to AI Agents with Accept Headers
#49Earlier quoted context omitted.
>On the caching impact, Simon Willison wrote: Wrong: https://developers.cloudflare.com/cache/concepts/vary/
Yes. This feature is brand new: https://developers.cloudflare.com/cache/changelog/ Simon wrote that in 2023: https://simonwillison.net/2023/Nov/20/cloudflare-does-not-co...
Thanks. I posted that comment before they had added support - https://news.ycombinator.com/item?id=48353325 - didn't know situation had changed.
But it's something I think developers should think about if they rely on caching. If it took Cloudflare this long to support this, there may be other systems which still don't.
Link to Roy Fielding comment:
https://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar...
Re: Serve Markdown to AI Agents with Accept Headers
#50I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…
Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy…
In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head.
But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know why anyone interested in this wouldn't just improve their HTML markup and leave it to the agent to do the rest. Convert to markdown or extract the useful HTML before handing it to model.