We've gone full circle: efficient meta languages back to inefficient and ambivalent natural language.
Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
41–50 of 73 posts
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#42I recommend shell-gpt[1] for anyone with access to the OpenAI API. It works surprisingly well considering how simple it is. Be sure to browse the examples in the README.
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#43very cool. But like many other uses of LLM it can hallucinate and/or produce a wrong result. For example I tried:
"gorilla dry run of brew upgrade"
And got a response that didn't work.
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#44It’s very sketchy that they use stderr and queries for training.
Don’t pass anything sensitive into this program!
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#45We've gone full circle: efficient meta languages back to inefficient and ambivalent natural language.
Agreed; this is somewhat useful for beginners but incredibly silly for professionals.
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#46Earlier quoted context omitted.
> It's like relying on a calculator. You still need mental math skills to know that 91 * 10 can't equal 2511. Similarly, when GPT starts hallucinating, it helps if you have a high sensitivity to smelling it out. Well, at least my calculators don't have this error rate GPT4 still does. Especially for seemingly simple things like a command flag, I have zero trust if GPT doesn't give me something that will eventually er…
how often are you using LLMs?
Somehow I knew that this question would come up, questioning the "progress" makes me a heretic.
So last 2-3 months I subscribed to ChatGPT4 (and much longer to Copilot), worked through most of the HN threads on tips and reviews, posts I could find on "prompt engineering" and have hundreds of sessions with ChatGPT4. So, I still might have missed something, but I think I have a rather good idea of what's going on.
1. It's rather good with understanding what I want. I can dump pretty much anything into it and give it certain rules (things we described years ago as "Google fu" until Google SERP became useless) and it will make something out of it.
2. It's a nice rubberduck to discuss things and get a broad overview on certain topics.
3. It's amazingly stupid, even if I ask it for its confidence, on the validity of its answers. It's like talking to a 8-year-old know-it-all: You have to fact check everything. If I confront it with the error, it even reacts like a 8-year old.
4. Initial responses for intentionally broad topics (summed up with "give me ansible yaml to deploy wireguard to N servers") are often times not working at all and after an hour of query-response you're better off reading ansible docs.
5. Initial responses for intentionally special topics (summed up with "what's the fastest algorithm to sort this given x, y, z and bla will never be A") it frequently comes up with good, sometimes surprisingly creative solutions.
All in all: Why oh why would I trade in correctness with a significant error rate ("hallucination" is a word from SV marketing hell) and debugging bullshit answers. Since debugging things is already a big drag in programming, I need things I can trust to build more things on top of them. If I can't trust 100% the "command" an LLM is generating, I'll never directly let it execute its code.
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#47I would encourage authors to update README with more representative examples.
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#48Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#49It’s very sketchy that they use stderr and queries for training. Don’t pass anything sensitive into this program!
Hey @linuxdude314 thank you for the comment. As we mentioned commands are executed solely with your explicit approval; And while we utilize queries and error logs (stderr) for model enhancement, we NEVER collect output data (stdout). This is a stronger guarantee than many of the other LLMs out there and our goal is to help this inform our research.
One of the reasons we open-sourced the front-end, is that if you would like to keep everything private, you can just clone the repo, comment out the logging, install it, and we will still honor and serve your queries if you hit our hosted end-point :) Let us know if there is anything more that we can do to make you comfortable in using our tool!
Re: Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs
#50very cool. But like many other uses of LLM it can hallucinate and/or produce a wrong result. For example I tried: "gorilla dry run of brew upgrade" And got a response that didn't work.
Thanks @ofermend, we believe that Gorilla will hallucinate lesser than other models but it's not zero yet! We will continue to reduce hallucination. Thanks for the feedback!