[flagged]
Serve Markdown to AI Agents with Accept Headers
81–90 of 97 posts
Re: Serve Markdown to AI Agents with Accept Headers
#82Earlier quoted context omitted.
Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy…
In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head. But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know w…
It also means you can’t easily know the set of formats that the server could respond with, which in turns makes e.g. archival more difficult.
Re: Serve Markdown to AI Agents with Accept Headers
#83Earlier quoted context omitted.
Presumably markdown uses far fewer tokens.
Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.
> HTML is about 5%-20% more tokens than a similar Markdown.
I'm not seeing this at all. I've got a small search engine I made that strips HTML back to Markdown for its full-text indexing. HTML is typically 10x bigger than the Markdown of the actual content, but that's because the majority of HTML out there is truly terrible.
I personally like HTML, and my own webpages are all hand-coded HTML. In that case, it's probably a closer ratio to what you describe. I'd suggest it's much higher than 20% more, but it's not likely more than double. But that's assuming someone paying attention to the efficiency of the HTML, and most people / websites just don't.
Markdown is even more readable without tools than HTML - it's essentially a plain text document - but I agree that HTML is better for actual semantic structure.
Re: Serve Markdown to AI Agents with Accept Headers
#84Earlier quoted context omitted.
If you do content negotiation, then it’s imperative to send “Vary: accept” in your response. CF and all other CDNs will automatically do the right thing when they see that header. Content negotiation still has its uses, but most of the time you’re better off using different endpoints.
Many CDNs do not support arbitrary Vary headers. Cloudfront, for example asks you to create a "Cache and Origin Request Policies" that gives you the ability to choose what headers are part of the request that gets sent to your origin. These will get added to the cache key, but it is a static list based on the request to the origin and not the response. Akamai is another case where Vary is harmful , from their docs [1…
It's often better to implement your caching logic at the Edge than just going with Vary support. Vanilla Vary support leads to cache dilution.
Accept (and other headers you might include in vary) may take many different values that you want to tie to a single cache entry.
Re: Serve Markdown to AI Agents with Accept Headers
#85Why would any website want to put extra effort to "give" their content as markdown to AI and get nothing in return ?
A question repeatedly asked by different people throughout these comments, but nobody answers.
Re: Serve Markdown to AI Agents with Accept Headers
#86Earlier quoted context omitted.
In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head. But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know w…
It also means you can’t easily know the set of formats that the server could respond with, which in turns makes e.g. archival more difficult.
That entirely depends on the server. A good solution would be to include Link headers in all responses:
Link: /some-page rel="canonical"
Link: /some-page.json rel="alternate" type="application/json"
Link: /some-page.html rel="alternate" type="text/html"
Link: /some-page.md rel="alternate" type="text/markdown"Re: Serve Markdown to AI Agents with Accept Headers
#87[flagged]
Re: Serve Markdown to AI Agents with Accept Headers
#88Why would any website want to put extra effort to "give" their content as markdown to AI and get nothing in return ?
An obvious example is if my product has an SDK and I'd like my customers to be able to script my product via their LLMs then I want to make my docs as AI-friendly as possible, since if their LLM picks my product's SDK rather than the competition, I make $$$.
Re: Serve Markdown to AI Agents with Accept Headers
#89Sounds like a nice idea for text-based browsers. Also, it makes a lot of advertising difficult.
OTOH, serving ads to mistrain AI content is an interesting business model.
Re: Serve Markdown to AI Agents with Accept Headers
#90Earlier quoted context omitted.
It also means you can’t easily know the set of formats that the server could respond with, which in turns makes e.g. archival more difficult.
That entirely depends on the server. A good solution would be to include Link headers in all responses: Link: /some-page rel="canonical" Link: /some-page.json rel="alternate" type="application/json" Link: /some-page.html rel="alternate" type="text/html" Link: /some-page.md rel="alternate" type="text/markdown"
That only works if the client looks at it. The current Claude fetch system does not.