Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use the Accept Header to Serve Markdown Instead of HTML to LLMs

skeptrune.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#21

Post by charcircuit »

krainboltgreene wrote: post #17
naet wrote: post #16

Earlier quoted context omitted.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

>Something you have no ability to manipulate.

What makes you think this?

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#22

Post by yawaramin »

krainboltgreene wrote: post #17
naet wrote: post #16

Earlier quoted context omitted.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

Why would a customer ask that? If I'm looking for something, why would I waste time with the worst version of it? I'd just go straight for the best.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#23

Post by yawaramin »

burcs wrote: post #5

Really cool idea Humans get HTML, bots get markdown. Two tiny tweaks I’d make... Send Vary: Accept so caches don’t mix Markdown and HTML. Expose it with a Link: …; rel="alternate"; type="text/markdown" so it’s easy to discover.

This person hypermedias

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#24

Post by Rohansi »

burcs wrote: post #5

Really cool idea Humans get HTML, bots get markdown. Two tiny tweaks I’d make... Send Vary: Accept so caches don’t mix Markdown and HTML. Expose it with a Link: …; rel="alternate"; type="text/markdown" so it’s easy to discover.

Would be nice for humans to get the markdown version too. Once it's rendered you get a clean page.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#25

Post by captn3m0 »

Rohansi wrote: post #24
burcs wrote: post #5

Really cool idea Humans get HTML, bots get markdown. Two tiny tweaks I’d make... Send Vary: Accept so caches don’t mix Markdown and HTML. Expose it with a Link: …; rel="alternate"; type="text/markdown" so it’s easy to discover.

Would be nice for humans to get the markdown version too. Once it's rendered you get a clean page.

I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#26

Post by skeptrune »

Kimitri wrote: post #19

The concept is called content negotiation. We used to do this when we wanted to serve our content as XHTML to clients preferring that over HTML. It's nice to see it return as I always thought it was quite cool.

Agreed! I love that such a tried and true web standard is making a comeback because of AI.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#28

Post by userbinator »

foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

And neither of those two ultimately help the humans who are actually looking for something. You have a finite amount of time to spend on optimising for humans, or for search engines (and now LLMs), and unfortunately many chose the latter and it's just lead to plenty of spam in the search results.

Yes, SEO can bring traffic to your site, but if your visitors see nothing of value, they'll quickly leave.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#29

Post by skeptrune »

klodolph wrote: post #27

I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?

It's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency.

Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#30

Post by klodolph »

skeptrune wrote: post #29
klodolph wrote: post #27

I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?

It's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency. Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.

This does not make any sense to me. Can you elaborate on this?

It seems “obvious” to me that if you have a tool which can request a web page, you can make it so that this tool extracts the main content from the page’s HTML. Maybe there is something I’m missing here that makes this more difficult for LLMs, because before we had LLMs, this was considered an easy problem. It is surprising to me that the addition of LLMs has made this previously easy, efficient solution somehow unviable or inefficient.

I think we should also assume here that the web site is designed to be scraped this way—if you don’t, then “Accept: text/markdown” won’t work.

Post reply on HN