Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serve Markdown to AI Agents with Accept Headers

acceptmarkdown.com

Post reply on HN
Print view View original

31–40 of 99 posts

Re: Serve Markdown to AI Agents with Accept Headers

#31

Post by slowin »

honr wrote: post #29
slowin wrote: post #20

Earlier quoted context omitted.

Presumably markdown uses far fewer tokens.

Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

> On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.

Re: Serve Markdown to AI Agents with Accept Headers

#32

Post by slowin »

simonw wrote: post #22
slowin wrote: post #20

Earlier quoted context omitted.

Presumably markdown uses far fewer tokens.

That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.

If it's one of many tool calls, I'd assume that less is more.

Re: Serve Markdown to AI Agents with Accept Headers

#33

Post by xienze »

honr wrote: post #29
slowin wrote: post #20

Earlier quoted context omitted.

Presumably markdown uses far fewer tokens.

Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

That's highly dependent on what sites you're visiting. Take a look around at a lot of modern sites, there's a sea of divs and spans. Markdown conversion helps LLMs a lot.

Re: Serve Markdown to AI Agents with Accept Headers

#34

Post by meindnoch »

k1m wrote: post #25

I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…

>On the caching impact, Simon Willison wrote:

Wrong: https://developers.cloudflare.com/cache/concepts/vary/

Re: Serve Markdown to AI Agents with Accept Headers

#35

Post by singpolyma3 »

k1m wrote: post #25

I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…

If CloudFlare isn't honouring vary: accept that's a pretty serious bug

Re: Serve Markdown to AI Agents with Accept Headers

#36

Post by singpolyma3 »

arjie wrote: post #28

Everything is determined by cloudflare. If they supported Gemini on the same hostname I’d serve it everywhere. If they supported text/plain and text/html everywhere I’d serve that. But they don’t cache per content encoding so I just do what works for them. They’re far too useful for me.

https://developers.cloudflare.com/cache/concepts/vary/

Re: Serve Markdown to AI Agents with Accept Headers

#37

Post by meindnoch »

collimarco wrote: post #19

What about clean, semantic HTML? It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now? There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

>What about clean, semantic HTML?

Which React package is this?

Re: Serve Markdown to AI Agents with Accept Headers

#38

Post by selcuka »

simonw wrote: post #22
slowin wrote: post #20

Earlier quoted context omitted.

Presumably markdown uses far fewer tokens.

That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.

They accept more tokens these days, but they are still more accurate with a shorter context [1].

[1] https://arxiv.org/abs/2307.03172

Re: Serve Markdown to AI Agents with Accept Headers

#39

Post by meindnoch »

Kuyawa wrote: post #18

Interesting... A cherry.jpg image/jpeg file is shown as an image by the browser A logo in svg like https://news.ycombinator.com/y18.svg too So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki Browsers should implement that asap and we will follow

It depends on one question: does Markdown support tags?

Re: Serve Markdown to AI Agents with Accept Headers

#40

Post by krapp »

meindnoch wrote: post #39
Kuyawa wrote: post #18

Interesting... A cherry.jpg image/jpeg file is shown as an image by the browser A logo in svg like https://news.ycombinator.com/y18.svg too So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki Browsers should implement that asap and we will follow

It depends on one question: does Markdown support tags?

Technically, Markdown supports all HTML tags, including script tags.

Practically, Markdown is more of a vibe than a spec and everyone just uses whatever subset makes sense to them.

Post reply on HN