Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use the Accept Header to Serve Markdown Instead of HTML to LLMs

skeptrune.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#31

Post by lelanthran »

captn3m0 wrote: post #25
Rohansi wrote: post #24

Earlier quoted context omitted.

Would be nice for humans to get the markdown version too. Once it's rendered you get a clean page.

I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.

> I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.

You can always do the markdown -> DOM conversion on the client. Sure, there's a bit of latency there, but it means easier deployment (no build step involving pandoc or similar).

Browser-native markdown support would be better though; you'd get ability to do proper contenteditable divs with bold, italic, etc done via markdown

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#32

Post by simonw »

klodolph wrote: post #27

I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?

Converting HTML into Markdown isn't particularly hard. Two methods I use:

1. The Jina reader API - https://jina.ai/reader/ - add r.jina.ai to any URL to run it through their hosted conversion proxy, eg https://r.jina.ai/www.skeptrune.com/posts/use-the-accept-hea...

2. Applying Readability.js and Turndown via Playwright. Here's a shell script that does that using my https://shot-scraper.datasette.io tool: https://gist.github.com/simonw/82e9c5da3f288a8cf83fb53b39bb4...

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#34

Post by pabs3 »

Kimitri wrote: post #19

The concept is called content negotiation. We used to do this when we wanted to serve our content as XHTML to clients preferring that over HTML. It's nice to see it return as I always thought it was quite cool.

Content negotiation is also good for choosing human languages, unfortunately the browser interfaces for it are terrible.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#35

Post by CGamesPlay »

foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

But software documentation is a prime example of when the incentives don't have any problems. I want my docs to be more accessible to LLMs, so more people use my software, so my software gets more mindshare, so I get more paying customers on my enterprise support plan.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#36

Post by xigoi »

captn3m0 wrote: post #25
Rohansi wrote: post #24

Earlier quoted context omitted.

Would be nice for humans to get the markdown version too. Once it's rendered you get a clean page.

I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.

Markdown is not standardized, so every browser would render the page differently and you’d get the same problems as with pre-standard HTML.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#37

Post by skeptrune »

CGamesPlay wrote: post #35
foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

But software documentation is a prime example of when the incentives don't have any problems. I want my docs to be more accessible to LLMs, so more people use my software, so my software gets more mindshare, so I get more paying customers on my enterprise support plan.

Oh hey, I work at Mintlify! We shipped this as a default feature for all of our customers.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#39

Post by skeptrune »

simonw wrote: post #32
klodolph wrote: post #27

I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?

Converting HTML into Markdown isn't particularly hard. Two methods I use: 1. The Jina reader API - https://jina.ai/reader/ - add r.jina.ai to any URL to run it through their hosted conversion proxy, eg https://r.jina.ai/www.skeptrune.com/posts/use-the-accept-hea... 2. Applying Readability.js and Turndown via Playwright. Here's a shell script that does that using my https://shot-scraper.datasette.io tool: https://gist…

I learned that the golang CLI[1] is the best through my work simplifying Firecrawl[2]. However, in this case I used one available through npmjs such that it would work with `npx` for the CF worker builds.

[1]: https://github.com/JohannesKaufmann/html-to-markdown

[2]: https://github.com/devflowinc/firecrawl-simple

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#40

Post by Vespasian »

krainboltgreene wrote: post #17
naet wrote: post #16

Earlier quoted context omitted.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

That is at most temporary. I expect within the next 5 year "partner products" and "LLM-optpmized content" will take the place of SEO.

The economic dynamics did not change and the methods will adapt.

Why wouldn't Google sell advertisers a prominent spot in the AI summary. That's their whole deal. Why wouldn't OpenAI do the same with (free) users.?

Post reply on HN