Live data from Hacker News

The Good and the Limitations of Github Copilot

blog.hrithwik.me

131–140 of 144 posts

Re: The Good and the Limitations of Github Copilot

#131
post #130
post #124

Earlier quoted context omitted.

> that generates code in a very context sensitive manner That's why I compared it to a markov chain aye In any case I've come to the conclusion my hater attitude is fueled by disappointment, so maybe it was the worst take from this whole thing. I'll avoid future copilot threads

markov chains model n grams.. this is really much better

Yes I was trying to be insulting at the time, which was wrong

Re: The Good and the Limitations of Github Copilot

#132
post #127

Earlier quoted context omitted.

Sure, it makes sense if you find something like `len_x` or `x_count` significantly less readable than `len(x)` and if you don't have to worry about resources. Can't say I've been there though. I did a quick test of this `reverse` function (which probably shouldn't exist in the first place) and, unsurprisingly, it became ~30% faster when `len(arr)` was only called once.

When I'm that resource constrained, I don't use Python. The changes necessary to make regular Python code performant defeat the goal of making it readable. There's a reason NumPy's innards aren't Python code.

Both sides of this "readability vs. performance" debate can be argued ad absurdum but that's not my intention. All I know is that I try to conserve resources no matter what level of the stack I'm working on and those generated snippets certainly don't!

Re: The Good and the Limitations of Github Copilot

#133

Earlier quoted context omitted.

This is exactly the problem. The regex issue isn’t that it’s not efficient, it’s that it’s wrong. Using this tool to generate code in a problem area you are not qualified to double-check and validate yourself is dangerous.

There's two problems: 1. generating code in the problem area (email address validation) which is pretty much a classic 'things programmers believe about' domain - https://haacked.com/archive/2007/08/21/i-knew-how-to-validat... 2. generating code in a programming idiom with which you are unfamiliar - which regex as a DSL is also a pretty classic example. I don't think most programmers are good at regex, I know I'm def…

aye - the biggest risk here is that rather than pulling in some standard lib for validating email addresses an engineer may use the copilot suggestion and validate it against a few simple test cases.

Curiously an attacker could probe services for use of the invalid suggestions that copilot generates....

Re: The Good and the Limitations of Github Copilot

#134

Earlier quoted context omitted.

This is exactly the problem. The regex issue isn’t that it’s not efficient, it’s that it’s wrong. Using this tool to generate code in a problem area you are not qualified to double-check and validate yourself is dangerous.

Very well put. Thank You. I know copilot is in alpha and will improve 100x but you will still need someone qualified to double check

ML applications face a frightening problem of diminishing returns on investment. The first prototype often happens in days or weeks, the next iteration months, after that years.

It's more analogous to clearing a foundation for a house by progressively picking up the boulders, then the rocks, then the grains of sand one at a time.

Re: The Good and the Limitations of Github Copilot

#135
post #12

Earlier quoted context omitted.

Does GitHub Copilot tell me which license the code it suggested has? If not it's a huge difference to code search engines.

Copilot is not a search engine. It synthesises code so it is unlicensed.

Copilot's api and suggestions could easily (and maybe was?) implemented as a SBQA style model. Using a search engine to find promising examples/context followed by a transformer model to synthesize the final output.

Attribution would clearly be required in such a search derived model.

Re: The Good and the Limitations of Github Copilot

#136

> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…

"Email address validation cannot be solved adequately with a regex" is something we need to start teaching somewhere. The RFC spec for emails is just way to permissive to make validating email addresses a winning move.

I swear I wind up having a battle over email validation at every company I go to. There is inevitably a business person that says "Well what about this site, they do it" and then I have to dig into whatever that site is actually doing and likely find a valid email address that breaks their validation to prove it.

And probably some junior dev (or senior who swears they did email validation flawlessly somewhere else and same story. I have to break their regex a bunch with valid emails they don't permit.

And of course then it's an uphill battle convincing them that what I'm using are in fact valid email addresses. Or you get the "Well no one ever actually does weird things in their email addresses so it's fine" or "gmail doesn't let me register that address so you're wrong"

Email is annoying.

Re: The Good and the Limitations of Github Copilot

#137

> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…

"Email address validation cannot be solved adequately with a regex" is something we need to start teaching somewhere. The RFC spec for emails is just way to permissive to make validating email addresses a winning move. I swear I wind up having a battle over email validation at every company I go to. There is inevitably a business person that says "Well what about this site, they do it" and then I have to dig into wha…

I like the HTML spec’s version, used for validation: https://html.spec.whatwg.org/multipage/input.html#valid-e-ma.... It allows all realistic inputs except for IP addresses (of dubious realism) and email address internationalisation (internationalised domain names are supported, but the local part is still currently stuck with ASCII, which is in keeping with its still-quite-limited support, though https://github.com/whatwg/html/issues/4562 progresses in fits and starts).

Re: The Good and the Limitations of Github Copilot

#138
post #119

Earlier quoted context omitted.

That’s actually roughly len(arr) times, not three, since two of the calls are inside the loop. But the far bigger red flag there is that that it doesn’t just use arr.reverse(), which does the same thing and is typically 8–10× as fast in some simple testing (assuming a list), or arr[::-1], which makes a shallow copy rather than modifying the object in-place. This matches what I’ve been seeing in code examples: Copilot…

> That’s actually roughly len(arr) times, not three, since two of the calls are inside the loop. You're right! I noticed it a few minutes ago and changed the wording accordingly. Thanks for pointing out how shockingly bad that algorithm actually is! :D > (...) the reverse function mutates its argument and returns it. Yeah, that mutate + return is confusing. It's also worth noting that, as a result of the mutation, th…

It's not shockingly bad to call len(arr) each time through the loop. It would be if we were talking about something like strlen() in C, but in Python it's just one more constant-time† operation each time through the loop, and not a very expensive one at that. Caching it in a local variable would still be better.

______

† Nothing is really constant-time in CPython, but it's pretty close.

Re: The Good and the Limitations of Github Copilot

#139
So far the most WTF thing I've gotten out of it is:

    import base64
    test = base64.b64decode("""SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t""".encode())
    print(test)
    # b"I'm killing your brain like a poisonous mushroom"
And the most odd thing:

    # The base URL for all API requests
    base_url = 'https://api.gdax.com/'
    # The base URL for all non-API requests (e.g. static content)
    base_url_static = 'https://static.gdax.com/'
Which are URLs that haven't been a thing for 2 years, I think and I can't find any code in github that uses them still.

Re: The Good and the Limitations of Github Copilot

#140

Earlier quoted context omitted.

"Email address validation cannot be solved adequately with a regex" is something we need to start teaching somewhere. The RFC spec for emails is just way to permissive to make validating email addresses a winning move. I swear I wind up having a battle over email validation at every company I go to. There is inevitably a business person that says "Well what about this site, they do it" and then I have to dig into wha…

I like the HTML spec’s version, used for validation: https://html.spec.whatwg.org/multipage/input.html#valid-e-ma... . It allows all realistic inputs except for IP addresses (of dubious realism) and email address internationalisation (internationalised domain names are supported, but the local part is still currently stuck with ASCII, which is in keeping with its still-quite-limited support, though https://github.com…

That's a good approach honestly.

I tend to just check for an @ and call it a day, validate it by emailing it and giving them a link to click if I need them to.

It's really the only way to ensure it's a valid and active address. People just don't want to build it.

Post reply on HN