Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To AI or not to AI

antropia.studio

Post reply on HN
Print view View original

41–50 of 90 posts

Re: To AI or not to AI

#41

Post by Helmut10001 »

latexr wrote: post #12
wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

> I am not a coder (much), but I have to wonder if my experience is common in the coding world? It is, yes. Surely someone will come and tell you it doesn’t happen to them, but all that tells you is that it ostensibly isn’t universal , but still common enough you’ll find no end of complaints. > Maybe there are ways to avoid this (and others) pitfall with prompting? Prompting can’t help you with things not in the trai…

Usually, in these edge cases, I go to the documentation page and dump all pages as Markdown into the AI tool (most often Gemini, due to token count). This context engeneering has helped a lot to get better answers. However, it also means I am consuming sometimes 1 Million tokens on relatively simple problems. Like recently, when I needed to solve a relativey simple but specific MermaidJS issue.

Re: To AI or not to AI

#42

Post by pletnes »

righthand wrote: post #35
wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

What is the cost (in tokens/$$$) of spending an hour restating questions to a chat bot vs typing `man rsync`?

Lower than the cost of a meatbag-office with heating, cooling and coffee for the time spent reading the manual.

Re: To AI or not to AI

#43

Post by erichocean »

wintermutestwin wrote: post #34
erichocean wrote: post #27

Earlier quoted context omitted.

Ask Gemini Pro 2.5 to build the rsync command and then give it the man page for your version of rsync. It should succeed the first time. Here's a command to copy the man page to the clipboard than you can immediately paste into aistudio (on a Mac): man rsync | col -b | pbcopy As a general rule, if you would need to look something up to complete a task, the AI needs the same information you do—but it's your job to pro…

So I paste the man page into llm and tell it to only give me parameters that are in that page? Even if it obeyed, it would still choke on how to exclude hidden MacOS kruft files from the copy...

No, you don't need to "tell it to only give me parameters that are in that page."

Here's the entire prompt:

    I need the rsync command to copy local files from `/foo/bar` to `~/baz/qux` on my `user@example.com` server.
    Exclude macOS cruft like `.DS_Store`, etc. Here's the man page:

Re: To AI or not to AI

#44

Post by wintermutestwin »

BryanLegend wrote: post #33
wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

I've recently been misled by ChatGPT a lot as well. I think it's the router. I'm on the free plan so I assume they're just being tight with the GPU cycles.

I am on a $20 plan and using the "thinking" version of 5.

Re: To AI or not to AI

#45

Post by nerdponx »

goalieca wrote: post #16
wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

> I tell it to validate each parameter against my version moving forward and it clearly doesn't do that. I would like an AI expert to weigh in on this point. I run into this a lot. It seems that LLMs, being language models and all, don't actually understand what i'm asking. Whenever i dive into the math, superficially, it kind of makes sense why they don't. But it also seems like transformers or some secret sauce is…

LLMs are next-token prediction models. They "understand" in that, if the previous 1000 tokens are such-and-such, then they emit a best guess at the 1001th token.

It "knows" what rsync is because it has a lot of material about rsync in the training data. However it has no idea about that particular version because it doesn't have much training data where the actual version is stated, and differences are elaborated.

What would probably produce a much better result if you included the man page for the specific version you have on your system. Then you're not relying on the model having "memorized" the relationship relationships among the specific tokens you are trying to get the model to focus on, instead just passing it all in as part of the input sequence to be completed.

It is absolutely astounding that LLMs work at all, but they're not magic, and some understanding of how they actually work can be helpful when it comes to using them effectively.

Re: To AI or not to AI

#46

Post by alain94040 »

wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

Instead of just saying: rsync on my system is version 3.2, have you tried copy/pasting rsync --help? In my experience, that would be enough for the AI to figure out what it needs to do and which arguments to use. I don't treat AI like an oracle, I treat it like an eager CS grad. I must give it the right information for the task.

Re: To AI or not to AI

#47

Post by nehal3m »

wintermutestwin wrote: post #34
erichocean wrote: post #27

Earlier quoted context omitted.

Ask Gemini Pro 2.5 to build the rsync command and then give it the man page for your version of rsync. It should succeed the first time. Here's a command to copy the man page to the clipboard than you can immediately paste into aistudio (on a Mac): man rsync | col -b | pbcopy As a general rule, if you would need to look something up to complete a task, the AI needs the same information you do—but it's your job to pro…

So I paste the man page into llm and tell it to only give me parameters that are in that page? Even if it obeyed, it would still choke on how to exclude hidden MacOS kruft files from the copy...

If you have trouble talking to an AI, how do you ever expect to merge with Neuromancer’s twin?

Re: To AI or not to AI

#48

Post by danielbln »

righthand wrote: post #35
wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

What is the cost (in tokens/$$$) of spending an hour restating questions to a chat bot vs typing `man rsync`?

Telling the agent to execute "man rsync" and synthesize the answer from there is probably the cheapest and most efficient option.

Letting some detached LLM fumble around for an hour is never the right way to go, and inversely sifting through the man page of rsync or fmmpeg or (God forbid) jq to figure out some arcane syntax isn't exactly a great use of anyone's time either, all things considered.

Re: To AI or not to AI

#49

Post by latexr »

wintermutestwin wrote: post #17
chrisweekly wrote: post #11

Earlier quoted context omitted.

Why involve an LLM at all, if you're looking up docs for a particular tool like rsync?

Lots of reasons! First off: where else do I go to learn this stuff? Man pages are reference for people who work in CLI all the time and not for virgin learners as they are are necessarily packed with the complete lexicon but with barely a thought to explaining real world examples of common tasks. There are a million Linux websites with the same versioning issues and inadequate explanations. I guess I could buy an ore…

> There are a million Linux websites with the same versioning issues and inadequate explanations.

Where do you think the LLM is getting its data from? At least on the website you can see the surrounding discussion and have a chance of finding someone with the same problem as you, or learning something tangential which will help you further down the line.

> With an LLM, I can get it to tell me what each parameter it suggests actually does and then I can ask it questions about that to further my understanding.

If it’s giving you wrong flags, why do you assume the explanations it gives you are accurate? LLMs can make those up just as well.

What you should do is verify the given flags on the man page. Not only will it clarify if they exist, it will also clarify if they’re what you’re looking for, and will likely even point to other relevant options.

Re: To AI or not to AI

#50

Post by BinaryIgor »

wintermutestwin wrote: post #9

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving…

All the time I find that if I know the stack and tools I am working in, it's faster to just write code on my own, manually; If I want to learn on the other hand, LLMs are quite useful - as long as you understand (or learn to) and validate the output
Post reply on HN