Just a thought that I have, wouldn’t it be better to block all robots and only to whitelist a select few? More AI bots are scraping now and in the future…
Author here.
I wish I could, but I bet most would just ignore robots.txt.
I Have Blocked OpenAI
11–20 of 57 posts
Re: I Have Blocked OpenAI
#12This is such a technopurist take. People who use LLM’s already know they can give wrong information. Your documentation won’t be able to cover every single possible contextual scenario that an LLM can help with. I think there are valid reasons to not allow OpenAI to spider you, but this is just a really silly one that feels pretty egotistical. People aren’t going to this guy saying “well OpenAI said your software wor…
They're only not doing that because my software is not common yet. But look at GitHub issues for any semi-famous project, and you'll see a lot of questions about misunderstandings, and that's before LLM's poisoned everything.
Re: I Have Blocked OpenAI
#13This is such a technopurist take. People who use LLM’s already know they can give wrong information. Your documentation won’t be able to cover every single possible contextual scenario that an LLM can help with. I think there are valid reasons to not allow OpenAI to spider you, but this is just a really silly one that feels pretty egotistical. People aren’t going to this guy saying “well OpenAI said your software wor…
> People who use LLM’s already know they can give wrong information
I think this is unfortunately much less true than expected... Lawyer using chatgpt.. teachers using chatgpt... even professors using chatgpt... as if its a source of truth.
Re: I Have Blocked OpenAI
#14I agree that LLMs are almost more likely than not to answer documentation questions wrong, to hallucinate methods that don’t exist, or just be silly. But the value I see in allowing LLMs to train on documentation is in the glue code that an LLM could (potentially!) generate. Documentation, even good docs, usually only answer the question “What does this method/class/general idea do?” Really good docs will come with s…
[deleted]
Re: I Have Blocked OpenAI
#15Does OpenAI also scan wayback machine? If it does and you are on that you may also wish to remove yourself.
Re: I Have Blocked OpenAI
#16> But here’s the problem: it will answer them wrong.
There is no way to know that, and even if it ends up being true, blocking openai will likely make the problem worse, e.g. the ai answers will be worse without access to the documentation.
Re: I Have Blocked OpenAI
#17This is such a technopurist take. People who use LLM’s already know they can give wrong information. Your documentation won’t be able to cover every single possible contextual scenario that an LLM can help with. I think there are valid reasons to not allow OpenAI to spider you, but this is just a really silly one that feels pretty egotistical. People aren’t going to this guy saying “well OpenAI said your software wor…
> People who use LLM’s already know they can give wrong information I think this is unfortunately much less true than expected... Lawyer using chatgpt.. teachers using chatgpt... even professors using chatgpt... as if its a source of truth.
There have been a few instances, sure, and they made headlines, but that was pretty early on when LLM behavior was not well understood. I think that fake citations (as the most obvious and well documented example) are a well understood problem now, and if you google “ChatGPT fake citation” you only get a few articles mostly referencing the same couple of cases from months ago. It doesn’t seem pervasive at all.
Re: I Have Blocked OpenAI
#18unpopular opinion: llm responses being wrong is still valuable to me since it gives me a better jumping off point to exploring than nothing at all. especially with something like coding that can easily be back-propagated due to something not compiling/not working as intended. could be harmful in other areas tho.
Author here. My docs will include tutorial links at the top, and those tutorials will focus on accomplishing common tasks. I believe that's a good jumping off point.
You can’t realistically cover every use case. With an LLM you can say something like “Make me a program that sets up an SDL window with the title ABCD that has a 396x224 RGB565 framebuffer and moves around a red square using the WASD keys by using a loop to fill in pixels in that framebuffer and then quits when it reaches the right edge of the screen” and it has a reasonable chance of making something that works or at least is easy to adjust into something that does. Just because sometimes it might not work the first time isn’t a good reason to try to stop people from using it entirely
Re: I Have Blocked OpenAI
#19I understand that some people don’t want their work to train AI. Personally I like that the work I publish is not completely useless as it is at least used to train LLMs.
Re: I Have Blocked OpenAI
#20> But here’s the problem: it will answer them wrong. There is no way to know that, and even if it ends up being true, blocking openai will likely make the problem worse, e.g. the ai answers will be worse without access to the documentation.
But if people come to me with problems, I can give them a link to that post and say, "GPTx does not know my stuff. You will want to read the docs yourself."