Live data from Hacker News

Use the Accept Header to Serve Markdown Instead of HTML to LLMs

skeptrune.com

51–60 of 64 posts

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#51
Maybe adopt the existing Gemini Protocol instead? It's already a nice very simple markdown-like.

https://toffelblog.xyz/blog/gemini-overview/ https://news.ycombinator.com/item?id=23730408

https://gemini.circumlunar.space/ https://news.ycombinator.com/item?id=23042424

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#52

Earlier quoted context omitted.

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

>Something you have no ability to manipulate. What makes you think this?

Because I built an LLM, I know how they work.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#53

Earlier quoted context omitted.

Neat up until the "customer ask" is "What, in X space, is the worst product you can purchase?" Something you have no ability to manipulate.

That is at most temporary. I expect within the next 5 year "partner products" and "LLM-optpmized content" will take the place of SEO. The economic dynamics did not change and the methods will adapt. Why wouldn't Google sell advertisers a prominent spot in the AI summary. That's their whole deal. Why wouldn't OpenAI do the same with (free) users.?

Because that’s not how LLMs work.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#54

Earlier quoted context omitted.

It's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency. Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.

This does not make any sense to me. Can you elaborate on this? It seems “obvious” to me that if you have a tool which can request a web page, you can make it so that this tool extracts the main content from the page’s HTML. Maybe there is something I’m missing here that makes this more difficult for LLMs, because before we had LLMs, this was considered an easy problem. It is surprising to me that the addition of LLMs…

If you have a website and you're optimizing it for GEO, you can't assume that the agents are going to have the glue. So as the person maintaining the website you implement as much of the glue as possible.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#55

Earlier quoted context omitted.

>Something you have no ability to manipulate. What makes you think this?

Because I built an LLM, I know how they work.

just add an arbiter layer on top for the possibility of advertising and modifying the output. not rocket science

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#56

Maybe adopt the existing Gemini Protocol instead? It's already a nice very simple markdown-like. https://toffelblog.xyz/blog/gemini-overview/ https://news.ycombinator.com/item?id=23730408 https://gemini.circumlunar.space/ https://news.ycombinator.com/item?id=23042424

FYI both the link to toffelblog and circumlunar.space are broken with ssl errors.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#57
post #16

Earlier quoted context omitted.

I do dev work for a marketing dept of a large company and there is a lot of talk about optimizing for LLMs/AI. Chatgpt can drive sales in the same way a blog post indexed by Google can. If a customer asks the AI what product can solve their problem and it replies with our product that is a huge win. If your business is SEO spam with online ads, chatgpt might eat it. But if your business is selling some product, chatg…

And what that means is the usefulness of LLms in recommending products is about to jump off a cliff.

This is what everybody should have expected.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#58

Earlier quoted context omitted.

And what that means is the usefulness of LLms in recommending products is about to jump off a cliff.

This is what everybody should have expected.

I think it’s going to be even worse - companies are going to go to ChatGPT with lawyers and say you are making false/unfair claims about our product. We should be able to give it this copy with correct information to consume.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#59
post #6

I think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.

If you are selling advertising, then I agree. However, if you are selling a product to consumers then no. Ask an LLM "What is the best refrigerator on the market." You will get various answers like:

> The best refrigerator on the market varies based on individual needs, but top brands like LG and Samsung are highly recommended for their innovative features, reliability, and energy efficiency. For specific models, consider LG's Smart Standard-Depth MAX™ French Door Refrigerator or Samsung's smart refrigerators with internal cameras.

Optimizing your site for LLM means that you can direct their gestalt thinking towards your brand.

Re: Use the Accept Header to Serve Markdown Instead of HTML to LLMs

#60
post #54

Earlier quoted context omitted.

This does not make any sense to me. Can you elaborate on this? It seems “obvious” to me that if you have a tool which can request a web page, you can make it so that this tool extracts the main content from the page’s HTML. Maybe there is something I’m missing here that makes this more difficult for LLMs, because before we had LLMs, this was considered an easy problem. It is surprising to me that the addition of LLMs…

If you have a website and you're optimizing it for GEO, you can't assume that the agents are going to have the glue. So as the person maintaining the website you implement as much of the glue as possible.

That sounds completely backwards. It seems, again, obvious to me that it would be easier to add HTML->markdown converters to agents, given that there are orders of magnitude more websites out there compared to agent.

If your agent sucks so bad that it isn’t capable of consuming HTML without tokenizing the whole damn thing, wouldn’t you just use an agent that isn’t such a mess?

This whole thing kinda sounds crazy inefficient to me.

Post reply on HN