Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serve Markdown to AI Agents with Accept Headers

acceptmarkdown.com

Post reply on HN
Print view View original

71–80 of 98 posts

Re: Serve Markdown to AI Agents with Accept Headers

#71

Post by stillpointlab »

chuckadams wrote: post #30
k1m wrote: post #25

I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…

If you do content negotiation, then it’s imperative to send “Vary: accept” in your response. CF and all other CDNs will automatically do the right thing when they see that header. Content negotiation still has its uses, but most of the time you’re better off using different endpoints.

Many CDNs do not support arbitrary Vary headers. Cloudfront, for example asks you to create a "Cache and Origin Request Policies" that gives you the ability to choose what headers are part of the request that gets sent to your origin. These will get added to the cache key, but it is a static list based on the request to the origin and not the response.

Akamai is another case where Vary is harmful, from their docs [1]:

    > As the content in response may be different for the same URL,  
    > Akamai  edge servers don't cache responses that include the Vary header, 
    > even if the content is cacheable by definition. The only exception is 
    > the case where the Vary header's value is Accept-Encoding and the
    > Content-Encoding header's value is br or gzip – edge servers cache 
    > such responses, applying the caching rules you defined in your property.
Cloudflare's docs do seem to indicate they support the Vary header as does Fastly. But one should read the docs of their CDN to find out the behavior. Do not assume Vary is supported.

https://techdocs.akamai.com/property-mgr/docs/rm-vary-header

Re: Serve Markdown to AI Agents with Accept Headers

#72

Post by troupo »

k1m wrote: post #50
9dev wrote: post #27

Earlier quoted context omitted.

Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy…

In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head. But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know w…

> I just don't like the idea of one URL being able to return different content.

It's different content representations. A text in a markdown file is conceptually the same content as the same text in HTML (or PDF).

Re: Serve Markdown to AI Agents with Accept Headers

#73

Post by troupo »

k1m wrote: post #48
collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

I agree. I think a lot of people here are assuming that the full HTML retrieved has to go into the LLM eating up tokens. But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM. Semantic HTML would make that easier.

> But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM.

There's no "agent". It's a few wrappers around API calls in a trenchcoat.

Re: Serve Markdown to AI Agents with Accept Headers

#74

Post by o_m »

slowin wrote: post #20
collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

Presumably markdown uses far fewer tokens.

Markdown isn't as expressive. Not all HTML content can be converted to Markdown without losing some of the semantics

Re: Serve Markdown to AI Agents with Accept Headers

#75

Post by blitzar »

qingcharles wrote: post #12
jiehong wrote: post #6

Earlier quoted context omitted.

Markdown ads?

Time magazine already serves their pages like this to agents with ads for the agents in them, IIRC.

Whats the sales agents conversion rate of selling agents to agents like?

Re: Serve Markdown to AI Agents with Accept Headers

#76

Post by fergie »

Unless I'm missing something here, this is all wrong. Markdown is OK, but for a variety of reasons, HTML is much better.

What you _should_ be doing is ensuring that your page works with screen readers. That simultaneously solves any kind of machine-readability issues and also makes your website accessible for the blind.

Re: Serve Markdown to AI Agents with Accept Headers

#77

Post by simonw »

slowin wrote: post #32
simonw wrote: post #22

Earlier quoted context omitted.

That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.

If it's one of many tool calls, I'd assume that less is more.

The trick there is to use a subagent to read the HTML page and extract the relevant information, than dumping all that HTML into your top-level session.

That's effectively using an LLM as an HTML to markdown converter, which is both absurdly wasteful and also surprisingly inexpensive (if you use a model like GPT-5.6 Luna.)

Re: Serve Markdown to AI Agents with Accept Headers

#78

Post by crimsonnoodle58 »

lekevicius wrote: post #11

I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption. I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.

You can see current support here:

https://acceptmarkdown.com/status

Re: Serve Markdown to AI Agents with Accept Headers

#79

Post by hnlmorg »

spockz wrote: post #17
hnlmorg wrote: post #14

Earlier quoted context omitted.

I don’t disagree with you in principle but there is a difference. The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.

Right. And today we have something similar with search engines going on where sites want to offer the full text to the engine for seo. But maybe they want to do the opposite to agent and poison the well. So human agents still get the “real” data and agents the “something plausible but not quite right” data. Or everything just disappears behind something like cloudflare or a paywall.

Funny enough, I was thinking about the SEO too when I wrote my comment.

The problem is that it’s in the sites interest to be SEOed. But it’s against their interests to be AI crawled.

Re: Serve Markdown to AI Agents with Accept Headers

#80

Post by Zardoz84 »

slowin wrote: post #20
collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

Presumably markdown uses far fewer tokens.

Not my problem. Currently my problem is the high traffic of bots that kills our web apps and don't respect robots.txt or meta tags . We are currently deploying Anubis.
Post reply on HN