Can I say - For RAG based system using LLM, will it be more effective to structure the source documents in this way?
Optimizing technical docs for LLMs
11–20 of 27 posts
Re: Optimizing technical docs for LLMs
#12Presumably all the structure and section headings that they recommend don't have to be rendered by a browser as visible to humans. The LLMs should be smart enough to understand HTML directives that don't add a lot of unnecessary visual structure.
Re: Optimizing technical docs for LLMs
#13Author here. Happy to answer any questions!
How do you measure any of these improvements?
Re: Optimizing technical docs for LLMs
#14This is useful stuff, but it is … familiar …
Re: Optimizing technical docs for LLMs
#15I love that writing LLM-friendly docs is just... writing good docs. There's a ton of overlap between accessibility work and preparing things to be used by LLMs. I wonder if an unintended side effect of this AI hype cycle is a huge investment in more accessible applications.
Exactly! It's very much garbage-in-garbage-out. One surprising (to me at least) benefit of hooking up an LLM to your docs is that it is actually a really useful way to find gaps in your docs. For example, when an LLM cannot answer a user question, there's a good chance it's because the answer is not documented anywhere.
What about when it answers with confabulation?
Re: Optimizing technical docs for LLMs
#16I love that writing LLM-friendly docs is just... writing good docs. There's a ton of overlap between accessibility work and preparing things to be used by LLMs. I wonder if an unintended side effect of this AI hype cycle is a huge investment in more accessible applications.
Just like how (it used to be) that writing good content would rank high in search results. But writing good content is hard. So we'll kid ourselves into writing good content for LLMs. Which is equally hard, but we'll feel like we're getting a leg up over everyone else -- who are all also doing the same thing.
It's unfortunate that people are more motivated to write for LLMs that are then used by humans than write for humans to begin with. Especially when the reason to use LLMs is because, on average, content is subpar making it difficult to find the good content.
Another case of Tragedy of the Commons Ruins Everything Around Me.
Re: Optimizing technical docs for LLMs
#17Author here. Happy to answer any questions!
What are the competing solutions for connecting github, docs to LLMs? Who are your main competitors?
Re: Optimizing technical docs for LLMs
#18Earlier quoted context omitted.
Exactly! It's very much garbage-in-garbage-out. One surprising (to me at least) benefit of hooking up an LLM to your docs is that it is actually a really useful way to find gaps in your docs. For example, when an LLM cannot answer a user question, there's a good chance it's because the answer is not documented anywhere.
What about when it answers with confabulation?
Confabulation should not be stable. Ie. you can generate answer let's say 3 times with different seed / non-zero temp and if it arrives at different answers you can categorize it as confabulation - which likely means, again, that the answer is not present and humans will likely also have different interpretations.
Re: Optimizing technical docs for LLMs
#19I felt most of this was just plain common sense. People read things by headings and sub headings. People look for relevant documentation for product x under product x not y. QA and code samples primes the LLM for what most developers like myself hunt for (a quick answer or a simple code snippet). The forum part got me. Seeing as forums tend to be variable in the quality and quantity of info. If the author(s) suggest…
You nailed it. All of above sources are also super helpful for LLMs, but you have to be careful about how to ingest/parse them. For example for a Discourse Forum only including questions that have been marked "Resolved" by an official team member can work quite well. Same goes for Discord/Slack forums and GitHub Discussions etc.
Re: Optimizing technical docs for LLMs
#20I love that writing LLM-friendly docs is just... writing good docs. There's a ton of overlap between accessibility work and preparing things to be used by LLMs. I wonder if an unintended side effect of this AI hype cycle is a huge investment in more accessible applications.
Just like how (it used to be) that writing good content would rank high in search results. But writing good content is hard. So we'll kid ourselves into writing good content for LLMs. Which is equally hard, but we'll feel like we're getting a leg up over everyone else -- who are all also doing the same thing. It's unfortunate that people are more motivated to write for LLMs that are then used by humans than write for…
Fair point. Although good writing for humans = good writing for LLMs and vice versa. So I'm hopeful this new excitement around AI for docs if anything will just encourage folks to put even more effort into writing great docs.