The web page does not load for me (I cannot reach the server), but the idea to serve Markdown according to the Accept header is possible, although it might be better to do that by changing the file name and adding a header (or a command) to indicate this possibility, so that there is a different file name in each case. Some already do this, allowing to change the file name for JSON or Markdown instead of HTML, and is helpful (there are also other common ways that you can change a URL to access it differently e.g. "blob" to "raw" in a git viewer, and I use these things sometimes). However, the reason should not be mainly for AI agents, even if it can be helpful for that purpose too.
Serve Markdown to AI Agents with Accept Headers
61–70 of 99 posts
Re: Serve Markdown to AI Agents with Accept Headers
#62This will be used to serve SEO slop. LLMs are already used as search engines and businesses really want to outrank competition by any means possible.
Re: Serve Markdown to AI Agents with Accept Headers
#63I agree with Roy Fielding on this: > It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know…
> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
Doing this with the Accept header is a bad idea, although I think CSS and JavaScripts (in web pages) is not a good solution to this either (they can often make it worse).
My way is the Scorpion conversion file, which must be downloaded explicitly by the end user and the end user must be allowed to override it with their own, and which tells it what to do when it receives a file that it does not recognize, based on the URL or the file type, such as: rewrite the URL, use a uxn program to convert it (to a format that you can use), use a uxn program to display it, etc. Something similar might be possible to add into WWW, by adding a "Interpreter:" response header into HTTP, perhaps using WebAssembly instead of uxn.
Re: Serve Markdown to AI Agents with Accept Headers
#64[dead]
Re: Serve Markdown to AI Agents with Accept Headers
#65How about allowing browsers to actually display markdown?
Re: Serve Markdown to AI Agents with Accept Headers
#66I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not g…
I think you are operating under the misconception that software developers are in their right mind
Re: Serve Markdown to AI Agents with Accept Headers
#67[flagged]
Re: Serve Markdown to AI Agents with Accept Headers
#68Why would any website want to put extra effort to "give" their content as markdown to AI and get nothing in return ?
Re: Serve Markdown to AI Agents with Accept Headers
#69I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not g…
i have built 2 html to document pipelines and read it back with pdftotext, and I can say that "just convert it yourself" step is the place where things are getting wrong.
For example Markdown has no support for the subscript and superscript in CommonMark or GFM and if you have 43 becomes 43 and water formula becomes H2O so in the output you get not the ugly but the wrong one. And no converter can fix it, because that's a target format which can't express it. extractors like Defuddle don't save from this also. and non of those extractors or converter throws you just get wrong text
Re: Serve Markdown to AI Agents with Accept Headers
#70I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not g…
> Who in their right mind would feed an LLM raw HTML in the first place??
I'd even say, is that even a problem ? If your website isn't a front end blob soup and just a relatively clean HTML file, your LLM is totally capable to handle it anyway.
And I'm still wondering myself which incentive do I have to serve my content to LLMs anyway.