Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use the Accept Header to Serve Markdown Instead of HTML to LLMs

skeptrune.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#11

Post by hahnbee »

nozzlegear wrote: post #8
skeptrune wrote: post #7

Earlier quoted context omitted.

This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.

I had a call with a new user for a SaaS product that I sell recently. During the call he mentioned that he found it by typing what he was looking for into Gemini, and it recommended my app. I don't do anything special for llms, and the public-facing part of the website has been neglected for longer than I like to admit, so I was delighted. I had never considered that AI could send new users to me rather than pull the…

> benevolent bullshit machine bestow a new user on me at the cost of nothing

that's awesome. i love this line.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#12

Post by foxfired »

skeptrune wrote: post #7
foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.

Yes, they show a tiny link behind a collapsed menu that very few people bother clicking. For example, my blog has gone from being prominently taking first spot on Google for some queries. Now with AI overviews, there is a sharp drop in traffic. However, it still showed higher impressions then ever. This means I'm appearing in search, even in AI overview, it's just that very few people click.

As of last week, impressions have also dropped. Maybe people not clicking on my links anymore is the result?

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#13

Post by bastawhiz »

gl-prod wrote: post #10
skeptrune wrote: post #7

Earlier quoted context omitted.

This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.

How many users do actually visit these links?

I usually do, as a data point.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#14

Post by skeeter2020 »

skeptrune wrote: post #7
foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.

and like Google - but much, much worse - they bring back enough content to keep users in the chat interface; they never visit your site.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#16

Post by naet »

foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can.

If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win.

If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatgpt might help you sell it.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#17

Post by krainboltgreene »

naet wrote: post #16
foxfired wrote: post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#18

Post by onion2k »

anabis wrote: post #15

OpenAI cookbook says LLMs understand XML better than Markdown text, so maybe that also? Although, it should be more specified and structured, but not HTML.

OpenAI cookbook says LLMs understand XML better than Markdown text.

Yes, for prompts. Given how little XML is out on the public internet it'd be surprising if it also applies to data ingestion from web scraping functions. It'd be odd if Markdown works better than HTML to be honest, but maybe Markdown also changes the content being served e.g. there's no menu, header, or footer sent with the body content.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#20

Post by Larrikin »

gl-prod wrote: post #10
skeptrune wrote: post #7

Earlier quoted context omitted.

This isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.

How many users do actually visit these links?

I've found it's extremely important to, because you will get some results that are already AI slop optimized to show in LLM research searches.
Post reply on HN