Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serve Markdown to AI Agents with Accept Headers

acceptmarkdown.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Serve Markdown to AI Agents with Accept Headers

#91

Post by tempfile »

phoghed wrote: post #15
Elfener wrote: post #8

Earlier quoted context omitted.

> But imagine for a second they did do that. You get the API, all the info is there. > Why do you need AI then? (from this HN comment, worth a read in full: https://news.ycombinator.com/item?id=43679585 )

> Let's say you want to plan a trip to Thailand with your family. You could use the fancy AI to do it for you, or you could build a stupid frontend with minimal natural language understanding. It's giving https://news.ycombinator.com/item?id=8863#9224

> It's giving

Except it isn't. Someone else could build that stupid frontend, and essentially create a better booking.com. Or you could ask the robot to make you a stupid frontend. It remains true that a well-designed MCP server goes wildly against the interests of sites like booking.com.

Re: Serve Markdown to AI Agents with Accept Headers

#93

Post by threatofrain »

meindnoch wrote: post #37
collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

>What about clean, semantic HTML? Which React package is this?

What’s semantic about HTML when in a long form document loving web we don’t even have a table of contents? Where’s the bibliography?

Re: Serve Markdown to AI Agents with Accept Headers

#94

Post by _heimdall »

9dev wrote: post #86
spider-mario wrote: post #82

Earlier quoted context omitted.

It also means you can’t easily know the set of formats that the server could respond with, which in turns makes e.g. archival more difficult.

That entirely depends on the server. A good solution would be to include Link headers in all responses: Link: /some-page rel="canonical" Link: /some-page.json rel="alternate" type="application/json" Link: /some-page.html rel="alternate" type="text/html" Link: /some-page.md rel="alternate" type="text/markdown"

For the LLM use, the challenge is that it will only discover those after first requesting and parsing the HTML version.

Maybe it will notice those, and maybe it will figure out the pattern for follow-up page requests, but there's no guarantee and it won't help the first request.

Re: Serve Markdown to AI Agents with Accept Headers

#95

Post by _heimdall »

k1m wrote: post #50
9dev wrote: post #27

Earlier quoted context omitted.

Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy…

In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head. But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know w…

The ideas is that the URL references the resource and the content type requested is only asking for that content in a different projection or representation.

The content at a URL should always match, the format in which its represented can be different based on the request. Its a bit like buying a book in hard copy or paperback, same book different format.

Re: Serve Markdown to AI Agents with Accept Headers

#96

Post by dubcanada »

slowin wrote: post #20
collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

Presumably markdown uses far fewer tokens.

Then convert from HTML to markdown before you convert to tokens? It's not rocket science, if markdown is "better" stripping all possible HTML tags and leaving just text, images also works. That is more then likely what any automated "serve markdown on the fly" would end up doing.

Re: Serve Markdown to AI Agents with Accept Headers

#97

Post by 9dev »

_heimdall wrote: post #94
9dev wrote: post #86

Earlier quoted context omitted.

That entirely depends on the server. A good solution would be to include Link headers in all responses: Link: /some-page rel="canonical" Link: /some-page.json rel="alternate" type="application/json" Link: /some-page.html rel="alternate" type="text/html" Link: /some-page.md rel="alternate" type="text/markdown"

For the LLM use, the challenge is that it will only discover those after first requesting and parsing the HTML version. Maybe it will notice those, and maybe it will figure out the pattern for follow-up page requests, but there's no guarantee and it won't help the first request.

Not necessarily. They could also send a HEAD request to the URL first, to see the headers only and decide on the available alternates.

I am well aware that few sites are taking that much care of their API in terms of HTTP features, but all of the problems discussed here have solid and battle-tested answers.

Re: Serve Markdown to AI Agents with Accept Headers

#98

Post by 9dev »

iamflimflam1 wrote: post #90
9dev wrote: post #86

Earlier quoted context omitted.

That entirely depends on the server. A good solution would be to include Link headers in all responses: Link: /some-page rel="canonical" Link: /some-page.json rel="alternate" type="application/json" Link: /some-page.html rel="alternate" type="text/html" Link: /some-page.md rel="alternate" type="text/markdown"

That only works if the client looks at it. The current Claude fetch system does not.

Should we let vibe-coded agent harnesses dictate protocol design now..?

On the flip side, I'd argue that the current centralisation of user agents (in the classical sense here) that benefit from programmatic content negotiation in form of a handful of harnesses like Claude or Codex is a great lever toward forcing the ecosystem to adopt better practices: If Anthropic added content negotiation as described in this thread to Claude, many sites would be incentivised to improve their web servers.

Re: Serve Markdown to AI Agents with Accept Headers

#99

Post by vidarh »

JimDabell wrote: post #51
lekevicius wrote: post #11

I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption. I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.

Claude Code’s been doing this for at least a year already and I assume Claude Cowork does it too as a result. That’s already millions of users, so it’s not “just a neat idea with no adoption”.

Anthropic's own documentation site also returns markdown (of sorts if you append ".md", and Claude Code knows it.

E.g.: https://code.claude.com/docs/en/overview.md

Post reply on HN