Live data from Hacker News

The Good and the Limitations of Github Copilot

blog.hrithwik.me

31–40 of 144 posts

Re: The Good and the Limitations of Github Copilot

#31
post #10

> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…

Copilot acts like a search engine, you search, you find, then you judge. It was never the case with search engines that you could just copy some code you found without verifying it. Also, it has the same copyright problems as if you used Google to find the code.

>Copilot acts like a search engine

No, it doesn't. If my understanding of it is correct, it's an autoencoder, then a few more bits of AI. The MINST dataset is a collection of hand written digits used in many early machine learning classes. Usually they are used to train a classifier, which returns the correct digit given an image. They can also be used to train an autoencoder, which will take an image in, compress it down to far fewer channels, and put out an image that quite closely matches the original.

Once you have an autoencoder, it is easer to input data, and train a neural network to do something with the compressed output. There is no way the autoencoder knows which samples were used to generate the resulting output, it's just optimized at compression.

Thus, Copilot isn't search. You could take the entire corpus it was trained on, and log all the compressed outputs. You could then take a given output before the autoencoder expands it back out, tell which few source code fragments were closest, but there are no guarantees.

TLDR; A far closer analogy: Copilot acts like a Comedian who has stolen a lot of jokes, and can't even remember where they came from.

Re: The Good and the Limitations of Github Copilot

#32
post #23

Earlier quoted context omitted.

I think Copilot should report the matching source URL to allow the user to visit the page and see the context and license. This move would also placate some copyright questions because it would be like searching StackOverflow or Github for inspiration. The problem of content attribution (exact and fuzzy match) has been studied before under the task of plagiarism detection for student essays. Funny thing is that a pla…

> I think Copilot should report the matching source URL That's the problem. The output of a GAN like Copilot usually can't be traced directly back to a single input.

A list sorted by probability would be better than nothing. Would a GAN like Copilot be able to provide that?

Re: The Good and the Limitations of Github Copilot

#34

Earlier quoted context omitted.

> I think Copilot should report the matching source URL That's the problem. The output of a GAN like Copilot usually can't be traced directly back to a single input.

A list sorted by probability would be better than nothing. Would a GAN like Copilot be able to provide that?

Not easily. It's a rather opaque process.

Re: The Good and the Limitations of Github Copilot

#35

A fun issue I keep hitting with Github Copilot in Python is that it's a coin flip whether it will give me a Python 3-style print statement or a Python 2.7 style print statement.

Given that no one should be _actively_ writing 2.7 code, and that GH is trying to monetize this, I would think they could retrain the main model that excludes 2.7 code, and then allow people who need 2.7 to "pay for supporting 2.7" just like I'm guessing folks stuck on old platforms always do

You raise a fascinating point about older platforms of other languages, too; Java has a super backward compat story, but woe be unto the coder who tries to name a variable "enum" nowadays

Re: The Good and the Limitations of Github Copilot

#38
post #10

> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…

Copilot acts like a search engine, you search, you find, then you judge. It was never the case with search engines that you could just copy some code you found without verifying it. Also, it has the same copyright problems as if you used Google to find the code.

Just like a real copilot in a car or an airplane shouldn't be trusted? Perhaps they should choose a different name then.

Re: The Good and the Limitations of Github Copilot

#39

> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…

This has been discussed at length several times before and the answer to why you use a regex like that to validate emails is because you aren't trying to validate against a standard, but against a subset of email formats. You want a "normal" simple email. No "+" domains etc. Because it doesn't matter if you annoy the 0.01% of your users that would be negatively affected by that, it's better to have their simple/canonical emails.

On the right side of the @ I agree completely, e.g. you must allow longer tlds. The regex is shit. But the same "simplification" thing would apply for IPv4: I'd probably want to have 4 groups of {0-9} even if a valid ipv4 address could be written in a lot more creative ways than that. The normal/simple/canonical way to write the address is a smaller scope than the set of allowed ways.

The regex to parse any valid email and the regex to parse the info I want, (perhaps from the user subset I want!) can be very different.

Edit: don't shoow the messenger - there is just zero chance you want a db that is more likely to contain user errors, has less valuable emails in it.

"Enter your email" doesn't mean "Enter a string that can be considered valid according to the RFC"!

No one cares whether "foo/baz=frob@example.com" is a valid email adress or not. In some cases you want RFC-compliant addresses, in which case it's a perfectly good idea to parse strictly to the spec. But in most cases you want a user identifier you know you can also contact with 100% certainty using some email SaaS. Or one you can cross reference to some other source. That's stricly a different purpose than parsing RFC compliant addresses. And allowing "foo@bar"@baz.com is just not a good idea.

Re: The Good and the Limitations of Github Copilot

#40

Earlier quoted context omitted.

A list sorted by probability would be better than nothing. Would a GAN like Copilot be able to provide that?

Not easily. It's a rather opaque process.

And besides, let's get real, nobody would look at the links - it defeats the purpose of copilot.
Post reply on HN