Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Gitgpt – Natural Language Git

github.com

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Show HN: Gitgpt – Natural Language Git

#51

Post by Oxidation »

williamcotton wrote: post #49
bqmjjx0kac wrote: post #48

Earlier quoted context omitted.

Extraordinary claims require extraordinary evidence. Given that GPT3 has 175 billion nodes, how would you even begin to support the claim that it never (or sufficiently rarely) does things that are surprising to humans?

If you're looking for academic research, this is a great place to start: https://arxiv.org/abs/2202.03629 But you can get a general feel by using ChatGPT. Open up a new conversation and ask it something like, "What is the capital of France?". Note the response. Open up a new conversation and note the response. Soon enough you should be able to see that the responses are far from random. You can use the OpenAI APIs di…

Ask it details about a little-documented event and it'll happily tell you plausible, but utterly false, lies, however.

Apparently the "early 2011 Bougainville earthquake" was magnitude 6.3, at a depth of 21.7km, on the 20th January and caused "widespread damage to buildings and infrastructure in the region, and triggered landslides that blocked roads and hampered rescue efforts".

It was actually on the 7th Feb, a 6.4 and at a depth of 415km. There were "no immediate reports of damage or injuries".

None of this is remotely surprising, considering it's a turbocharged statistical model and it probably ingested a few words about it at most, out of billions and billions, but somewhere along the line from "famous" to "footnote" subjects, it will segue into complete fiction.

Re: Show HN: Gitgpt – Natural Language Git

#52

Post by klabb3 »

danwee wrote: post #9
gdcbe wrote: post #4

Not sure what value it adds though. Isn’t it easier to type the cmds then to write all that? If there’s a good product in it I’m afraid your examples aren’t selling it :S Was first thinking it could have value for when you forget a cmd or when learning, but I think if you can come up with those technical instructions you can also remember the cmds …

A colleague of mine didn't know that it was possible to revert a commit without commiting (so, leaving the reverted changes in the staging area). In such ocassions you either a) search in google, b) read the manual, c) ask a colleague But with a command line tool like Gitgpt I guess one could just do: > gitpgt "revert commit without commiting" and the command line tool would answer "git revert --no-commit COMMIT_ID_H…

> A colleague of mine didn't know that it was possible to revert a commit without commiting […]

> But with a command line tool like Gitgpt I guess one could just do:

>> gitpgt "revert commit without commiting"

This is a classic post-hoc curse of knowledge fallacy. I don’t blame you, this is something that people with really analytically strong minds struggle with, such as the fine folks who made voice assistants for Google, Apple and Amazon.

Asking someone who is trying to be overly helpful to do something you don’t know if they can do is at best meaningless (if it can’t do any harm) and at worst a really bad idea (if it can eg screw up your git repo). In your friend’s mind, it’s very possible that git can’t do this, and then who knows what the GPT will try to do instead to please you? This is the main crux of the problem.

Even if they think it’s possible but they don’t know the command, they don’t know if “revert without committing” is a meaningful sentence, perhaps git has another abstraction model and uses different terms. This is the spirit behind the expression “asking the right question takes as much skill as giving the right answers”.

GPT can certainly help explaining, summarizing etc, but it has a very limited ability to say no, and more generally pointing out when your mental model is wrong. I’ve had long complex “design reviews” with ChatGPT – it’s really good at going along with your assumptions, but it lacks the “spine” to tell you when your logic isn’t coherent, or that you have mutually exclusive goals, or that you’ve strayed “too far”, whatever that means.

Someone should write a long blog post about this paradox of helpfulness, I assure you it’s a real thing!

Re: Show HN: Gitgpt – Natural Language Git

#53

Post by OddMerlin »

cowsup wrote: post #22
mellosouls wrote: post #18

Earlier quoted context omitted.

Not sure what value it adds though. Isn’t it easier to type the cmds then to write all that? It's pretty obvious it's intended use cases include memory prompts for experienced users and translations from English requirements for inexperienced users. It's even in the OP: use natural language to get git to do what you want.

But when the prompt is: > add .gitignore commit with msg adding ignore and push That's not very compelling. The main selling point of AI is using simple inputs to achieve advanced outputs that match what the user wanted. If I can say "Commit my .gitignore file," and then it runs git add .gitignore git commit -m "Adding .gitignore" git push [y/N] y Then that's one thing. But this looks like you still have to specify t…

OP here. I love this feedback. I agree this is definitely something I should work towards.

Keeping the commands as succinct as possible as well as "natural language" as possible. Even the word commit might be too git-y?

Thanks for the feedback.

Re: Show HN: Gitgpt – Natural Language Git

#54

Post by williamcotton »

Oxidation wrote: post #51
williamcotton wrote: post #49

Earlier quoted context omitted.

If you're looking for academic research, this is a great place to start: https://arxiv.org/abs/2202.03629 But you can get a general feel by using ChatGPT. Open up a new conversation and ask it something like, "What is the capital of France?". Note the response. Open up a new conversation and note the response. Soon enough you should be able to see that the responses are far from random. You can use the OpenAI APIs di…

Ask it details about a little-documented event and it'll happily tell you plausible, but utterly false, lies, however. Apparently the "early 2011 Bougainville earthquake" was magnitude 6.3, at a depth of 21.7km, on the 20th January and caused "widespread damage to buildings and infrastructure in the region, and triggered landslides that blocked roads and hampered rescue efforts". It was actually on the 7th Feb, a 6.4…

Sure, but that is a predictable kind of result, not “rm -rf /“.

Re: Show HN: Gitgpt – Natural Language Git

#56

Post by AlexAltea »

williamcotton wrote: post #40
AlexAltea wrote: post #27

That looks cool, though I'd be very concerned about the possibility about ChatGPT "hallucinating" a `rm -rf /` or equivalent.

When LLMs “hallucinate” they don’t do so in random ways like somehow deciding to “rm -rf /“. They do so in predictable ways.

Sure, perhaps my example was too extreme. What about:

  $ gitgpt commit files with msg cleaning repo files and push

  git commit -m msg
  git clean -fdx
  git push
Without `git-add`, 1st one is a NO-OP, 2nd is a destructive action, 3rd is a NO-OP. All vaguely related to the topic at hand, "hallucinating" such a destructive action seems at least plausible.

Re: Show HN: Gitgpt – Natural Language Git

#57

Post by williamcotton »

AlexAltea wrote: post #56
williamcotton wrote: post #40

Earlier quoted context omitted.

When LLMs “hallucinate” they don’t do so in random ways like somehow deciding to “rm -rf /“. They do so in predictable ways.

Sure, perhaps my example was too extreme. What about: $ gitgpt commit files with msg cleaning repo files and push git commit -m msg git clean -fdx git push Without `git-add`, 1st one is a NO-OP, 2nd is a destructive action, 3rd is a NO-OP. All vaguely related to the topic at hand, "hallucinating" such a destructive action seems at least plausible.

Yup, that is something more realistic.

The GitHub Copilot CLI tool that is in beta, along with other tools, will show you the command and then make you manually choose to run, redo or abort.

In practice I have yet to accidentally perform any destructive operations. This makes intuitive sense because those seem like unlikely completions for a model that has been fine-tuned on “helpful cli recommendations”!

Re: Show HN: Gitgpt – Natural Language Git

#58

Post by bqmjjx0kac »

williamcotton wrote: post #49
bqmjjx0kac wrote: post #48

Earlier quoted context omitted.

Extraordinary claims require extraordinary evidence. Given that GPT3 has 175 billion nodes, how would you even begin to support the claim that it never (or sufficiently rarely) does things that are surprising to humans?

If you're looking for academic research, this is a great place to start: https://arxiv.org/abs/2202.03629 But you can get a general feel by using ChatGPT. Open up a new conversation and ask it something like, "What is the capital of France?". Note the response. Open up a new conversation and note the response. Soon enough you should be able to see that the responses are far from random. You can use the OpenAI APIs di…

Thanks for the link! I don't think that really addresses my concern, though.

My point is that these LLMs are basically incredibly large programs that defy analysis with our current tools. Sure, I can poke it a few times and see that it usually does what I want, but that's not the same as saying it never goes off the rails.

If it does something crazy like post my bank login online, even only once in a billion times, that's still orders of magnitude higher than I'm willing to accept.

Re: Show HN: Gitgpt – Natural Language Git

#59

Post by Oxidation »

williamcotton wrote: post #54
Oxidation wrote: post #51

Earlier quoted context omitted.

Ask it details about a little-documented event and it'll happily tell you plausible, but utterly false, lies, however. Apparently the "early 2011 Bougainville earthquake" was magnitude 6.3, at a depth of 21.7km, on the 20th January and caused "widespread damage to buildings and infrastructure in the region, and triggered landslides that blocked roads and hampered rescue efforts". It was actually on the 7th Feb, a 6.4…

Sure, but that is a predictable kind of result, not “rm -rf /“.

Some flavour of "git gc" after your reset is far more likely to crop up and ruin your day, that's true.

As long as you stay on the statistical beaten path (i.e. you're asking about Paris), you will probably be fine, indeed. Probably. Stochastic bugs are always the most fun anyway.

Re: Show HN: Gitgpt – Natural Language Git

#60

Post by williamcotton »

bqmjjx0kac wrote: post #58
williamcotton wrote: post #49

Earlier quoted context omitted.

If you're looking for academic research, this is a great place to start: https://arxiv.org/abs/2202.03629 But you can get a general feel by using ChatGPT. Open up a new conversation and ask it something like, "What is the capital of France?". Note the response. Open up a new conversation and note the response. Soon enough you should be able to see that the responses are far from random. You can use the OpenAI APIs di…

Thanks for the link! I don't think that really addresses my concern, though. My point is that these LLMs are basically incredibly large programs that defy analysis with our current tools. Sure, I can poke it a few times and see that it usually does what I want, but that's not the same as saying it never goes off the rails. If it does something crazy like post my bank login online, even only once in a billion times, t…

You’re basically asking me to prove to you that I can’t fly.

I will say it like this: it is highly improbable that I can fly. I cannot come up with a way to prove it to you. There is some sort of epistemic miscalculation going on if you operate under the assumption that I might be able to fly.

Post reply on HN