The Opposite of Documentation is Superstition (2020)
buttondown.com
The Opposite of Documentation is Superstition (2020)
1–10 of 87 posts
Re: The Opposite of Documentation is Superstition (2020)
#2I have also used the phrase "tribal knowledge" in the documentation-vacuum context.
Re: The Opposite of Documentation is Superstition (2020)
#3I have also used the phrase "tribal knowledge" in the documentation-vacuum context.
There could be a business to be made in capturing and disseminating tribal knowledge for organizations. Even orgs with the best documentation don't have a great mechanism to capture and make available tribal knowledge which is un-authoritative, sometimes conflicting, and constantly changing.
In the past I made an effort to address this by creating a unofficial but quickly searchable document repository documenting every question I have gotten from a teammate by either answering it or pointing to where the authoritative answer could be found.
What I found was the following:
1. Questions are repetitive, you can quickly build a base of tribal knowledge and point people to the answer. 2. If the document tool is quick and efficient, team members will generally check it first instead of asking. 3. Sometimes document tribal knowledge conflicts with someone else's tribal knowledge. This is a great opportunity to hash out the more correct answer and update the document. 4. Tribal knowledge gets stale fast and needs constant curating. 5. The less friction to find an answer, eg fast access and fast/accurate searching, greatly increases success.
NB. GPT is a perfect front end for such an effort.
Re: The Opposite of Documentation is Superstition (2020)
#4Seems to me documentation can also be superstition if it's out of date.
I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy
Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
Re: The Opposite of Documentation is Superstition (2020)
#5Seems to me documentation can also be superstition if it's out of date. I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
Maybe LLMs make docs irrelevant if they are just generating static pages or doc strings. Why spend the time generating something static when everyone has their own LLMs they can just ask directly? Anything you can generate as docs someone can generate themselves anyways.
Re: The Opposite of Documentation is Superstition (2020)
#6This is an interesting case, since the pigeon study is about what happens when the underlying process is random.
But if the shape drawing process isn't random, I think the author's experience of feeling unable to articulate the rules AND gravitating to a set of behaviors is a good example of procedural memory (implicit vs explicit).
Explicit rules would probably help speed things up, though!
Re: The Opposite of Documentation is Superstition (2020)
#7Seems to me documentation can also be superstition if it's out of date. I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
Maybe LLMs make docs irrelevant if they are just generating static pages or doc strings. Why spend the time generating something static when everyone has their own LLMs they can just ask directly? Anything you can generate as docs someone can generate themselves anyways.
We actually ran this train of thought with our startup right now. The problem? Not everybody knows what questions to ask. For non-power users it’s usually better to present digestible information and create docs that answer questions they didn’t even know they had.
Re: The Opposite of Documentation is Superstition (2020)
#8Seems to me documentation can also be superstition if it's out of date. I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
Maybe LLMs make docs irrelevant if they are just generating static pages or doc strings. Why spend the time generating something static when everyone has their own LLMs they can just ask directly? Anything you can generate as docs someone can generate themselves anyways.
I mean, LLMs can be regenerated as needed.
With every major release, you might regenerate the documentation. Then, you don't need to have everyone taking time to ask the same questions, reducing the amount of times you need to call the LLM. And thereby you can use a more expensive LLM, like o1 to generate the documentation where you wouldn't want to use it normally
Re: The Opposite of Documentation is Superstition (2020)
#9Seems to me documentation can also be superstition if it's out of date. I wonder if LLMs can generate docs at any reasonable level of usefulness and accuracy Speaking of which, I just tried using ChatGPT to create a React app with typescript and rollup. What it generated didn't work (got errors). Of course I could tell it the errors and it fixed them but it suggests they aren't yet ready to write accurate docs
Why would you bring ML up? The entire example problem brought up by TFA is because the ink-to-shape transform likely uses some ML behind the scenes (plain interpolation likely) , and as consequence, it is an _indocumentable_ blackbox even if you wanted to document it. Superstition is the only way you can make it work.
In fact, "superstition" very neatly defines most prompt engineering for LLMs.
Re: The Opposite of Documentation is Superstition (2020)
#10I have also used the phrase "tribal knowledge" in the documentation-vacuum context.
There could be a business to be made in capturing and disseminating tribal knowledge for organizations. Even orgs with the best documentation don't have a great mechanism to capture and make available tribal knowledge which is un-authoritative, sometimes conflicting, and constantly changing. In the past I made an effort to address this by creating a unofficial but quickly searchable document repository documenting ev…
I would have loved to train an LLM on my old technical blog.
In my last job, I used a blog to write informal reports. I set pretty loose rules for the blog for my team. We would post meeting minutes, outage postmortems, scripts, tricky one-liners, general lessons learned, and so on. We used SharePoint, which is kinda meh but it works. The "Alert Me" feature in SharePoint made it easy to subscribe my teammates to the blog via email, so you got the benefits of email (push notifications) without the drawbacks (employee mailboxes are unsearchable once they leave).
I believe email contributes to this tacit ("tribal") knowledge that the OP names. I think my blog went a long way to counter this effect. A major goal in our blog posts was to capture the "why" behind our decisions. Often, we fail to remember the nuances of our own decisions in hindsight. I always thought it was important to name our present priorities and known-unknowns.