Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serve Markdown to AI Agents with Accept Headers

acceptmarkdown.com

Post reply on HN
Print view View original

51–60 of 99 posts

Re: Serve Markdown to AI Agents with Accept Headers

#51

Post by JimDabell »

lekevicius wrote: post #11

I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption. I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.

Claude Code’s been doing this for at least a year already and I assume Claude Cowork does it too as a result. That’s already millions of users, so it’s not “just a neat idea with no adoption”.

Re: Serve Markdown to AI Agents with Accept Headers

#52

Post by k1m »

joshum97 wrote: post #21

I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not g…

Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version.

The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.

I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.

Re: Serve Markdown to AI Agents with Accept Headers

#54

Post by chuckadams »

singpolyma3 wrote: post #35
k1m wrote: post #25

I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…

If CloudFlare isn't honouring vary: accept that's a pretty serious bug

They just started fully honoring it last month. I guess my comment above was overly optimistic.

Re: Serve Markdown to AI Agents with Accept Headers

#55

Post by russ_j »

lekevicius wrote: post #11

I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption. I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.

Also it's not that hard at crawl time to transform HTML into markdown before passing to the LLM.

Re: Serve Markdown to AI Agents with Accept Headers

#56

Post by arjie »

singpolyma3 wrote: post #36
arjie wrote: post #28

Everything is determined by cloudflare. If they supported Gemini on the same hostname I’d serve it everywhere. If they supported text/plain and text/html everywhere I’d serve that. But they don’t cache per content encoding so I just do what works for them. They’re far too useful for me.

https://developers.cloudflare.com/cache/concepts/vary/

Quite helpful, I’ll see if I can serve text/plain.

Re: Serve Markdown to AI Agents with Accept Headers

#57

Post by Semiapies »

joshum97 wrote: post #21

I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not g…

Who in their right mind would feed any random LLM crawler their website's content?

Re: Serve Markdown to AI Agents with Accept Headers

#58

Post by honr »

slowin wrote: post #31
honr wrote: post #29

Earlier quoted context omitted.

Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

> On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown. This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.

No, it's just that I wasn't very clear.

HTML I can view in any browser / webview / etc. Good markdown viewers are fewer / more special, or end up translating md to html for display.

And by composing, I didn't mean writing by hand. We are talking about prompting, right? Or that is what I thought we are talking about. Composing HTML "components" into a final prompt HTML is easier than composing markdown snippets into the final prompt. That is because with HTML there are several ergonomic libraries to parse HTML to AST and to format AST back to HTML. The libraries (for parsing to AST and back to strings) are more limited with markdown.

Re: Serve Markdown to AI Agents with Accept Headers

#59

Post by honr »

xienze wrote: post #33
honr wrote: post #29

Earlier quoted context omitted.

Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

That's highly dependent on what sites you're visiting. Take a look around at a lot of modern sites, there's a sea of divs and spans. Markdown conversion helps LLMs a lot.

Aah, I thought we are talking about prompting or providing information to AI agents in either html or md form, and comparing the two.

Assuming that is what we are talking about, HTML is easier to work with than Markdown, unless you are writing it by hand. That is, composing semantic HTML is more ergonomic than composing a Markdown formatted document from components / snippets, programmatically. The libraries are just better and more versatile in most programming languages. Typically you go from HTML or Markdown to AST, then you compose them to end up with the final tree, then you format the tree to HTML or Markdown. LLMs treat them basically identically (context in HTML or context in Markdown), so I have ended up forming complex prompts / context parts using HTML.

Post reply on HN