Or one can just use semantic HTML; it's easy enough to convert semantic HTML into markdown with a tool like pandoc. That would also help screen readers, browser "reader modes", text-based web browsers, etc.
Use the Accept Header to Serve Markdown Instead of HTML to LLMs
41–50 of 64 posts
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#42Earlier quoted context omitted.
This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.
Yes, they show a tiny link behind a collapsed menu that very few people bother clicking. For example, my blog has gone from being prominently taking first spot on Google for some queries. Now with AI overviews, there is a sharp drop in traffic. However, it still showed higher impressions then ever. This means I'm appearing in search, even in AI overview, it's just that very few people click. As of last week, impressi…
Maybe it's about adding knowledge to LLMs, and not how many people read your website? I would be very happy if i had a simple way to get my insights, knowledge and best practices into the next version of an LLM so I have a way to improve it.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#43I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.
I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…
And what that means is the usefulness of LLms in recommending products is about to jump off a cliff.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#44Earlier quoted context omitted.
I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.
> I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML. You can always do the markdown -> DOM conversion on the client. Sure, there's a bit of latency there, but it means easier deployment (no build step involving pandoc or similar). Browser-native markdown support would be better though; you'd get ability to do proper contenteditable…
To get broad support from the server side, you’ll need to showcase high browser support. We need Wordpress and Wikipedia and Ghost to support this, and that won’t happen without native browser support.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#45Earlier quoted context omitted.
I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.
Markdown is not standardized, so every browser would render the page differently and you’d get the same problems as with pre-standard HTML.
Browsers can take standards position on CommonMark extensions and decide on a baseline that goes into the W3C spec? It will just converge on the lowest common denominator and that’s good enough for the vast majority of content reading usecases.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#46I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?
It's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency. Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.
I don't think it's about including this as a tool, just as general preprocessing before the agent even gets the text.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#47Earlier quoted context omitted.
It's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency. Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.
I don't think it's about including this as a tool, just as general preprocessing before the agent even gets the text.
Well that's what I implemented. There are markdown docs for every HTML file and the proxy decides to serve either markdown or HTML based on the Accept header.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#48I don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?
Converting HTML into Markdown isn't particularly hard. Two methods I use: 1. The Jina reader API - https://jina.ai/reader/ - add r.jina.ai to any URL to run it through their hosted conversion proxy, eg https://r.jina.ai/www.skeptrune.com/posts/use-the-accept-hea... 2. Applying Readability.js and Turndown via Playwright. Here's a shell script that does that using my https://shot-scraper.datasette.io tool: https://gist…
A lightweight alternative to Playwright, which starts a browser instance, is using an HTML parser and DOM implementation like linkedom.
This is much cheaper to run on a server. For example: https://github.com/ozanmakes/scrapedown
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#49Earlier quoted context omitted.
> I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML. You can always do the markdown -> DOM conversion on the client. Sure, there's a bit of latency there, but it means easier deployment (no build step involving pandoc or similar). Browser-native markdown support would be better though; you'd get ability to do proper contenteditable…
To get broad support from the server side, you’ll need to showcase high browser support. We need Wordpress and Wikipedia and Ghost to support this, and that won’t happen without native browser support.
> We need Wordpress and Wikipedia and Ghost to support this, and that won’t happen without native browser support.
It can. Unlikely but possible. A good first step would be to have a well-written web component to be used like this: `...`, with no support at all for a build-step. The .js file implementing this should be included directly in the ``.
If that gets traction (unlikely, but possible) then the standards would sooner or later introduce a tag native to the browser that does the same thing.
Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs
#50Earlier quoted context omitted.
I don't think it's about including this as a tool, just as general preprocessing before the agent even gets the text.
Well that's what I implemented. There are markdown docs for every HTML file and the proxy decides to serve either markdown or HTML based on the Accept header.
I think GP meant on the client, i.e. agent side. As in, you could deploy this kind of proxy in a forward/non-reverse way inside the agent system, so the LLM always gets markdown, regardless of what the site supports.
There is no real reason to pass HTML with tags and all to the LLM - you can just strip the tags beforehand.