You know they are hitting something interesting if all of hackernews gets mad haha
The Good and the Limitations of Github Copilot
111–120 of 144 posts
Re: The Good and the Limitations of Github Copilot
#112Earlier quoted context omitted.
this is the worse take I've seen so far
Could you explain why or do you just want to feel superior? Both good choices honestly but I'd love to know your actual thoughts If I'm wrong tell me why and I'll happily reevaluate my opinion, promise :)
Are you aware co-pilot is still in preview?
It's a bit harsh to make sweeping statements along the lines of 'it's just a fancy markov bot' based on a few well-publicised glitches in a technical preview.
I assume you have built something surpassing the scope and ambition of co-pilot before, not just some armchair tech lead throwing shade.
Re: The Good and the Limitations of Github Copilot
#113Earlier quoted context omitted.
> Using this tool to generate code in a problem area you are not qualified to double-check and validate yourself is dangerous. I would like this message to be amplified as much as possible. Never write code you do not understand. I am excited about copilot, but also wary of the programming culture these tools will bring in. Businesses, especially body-shopping companies will want to deliver as much using tools in thi…
Isn't "Code you don't understand" the definition of AI/ML?
> Isn't "Code you don't understand" the definition of AI/ML?
We don't need to understand the process to evaluate the output in this case. Bad code is bad code no matter who/what wrote it.
Re: The Good and the Limitations of Github Copilot
#114> Developers with experience can definitely handle this out but what if a newbie directly starts with the help of AI, he will spend more time on stack overflow than writing actual code. (oh wait that's how most of the developers are lmao)
Spontaneous Ask HN: How much truth is there in this "devs be copy-pasting from SO all day" trope?
Personally, I have used SO quite heavily in its early years, circa 2010-2014, including posting my own questions and sometimes posting answers to others' questions. But now, I don't use it actively anymore. Sure, when I search for a concrete question and SO happens to be in the search results, it's sometimes a valuable resource. But it's not the go-to for programming questions that it once was for me.
I'm honestly not sure if that's indicative of my own growth as a developer, or caused by outside factors. I have a vague feeling that developer documentation in general got better in the last decade, at least for the technologies that I'm using... but then again it could be also a sign of personal growth that I'm more comfortable with the upstream documentation. Finding answers for webdev questions on MDN is another sort of game than finding answers for webdev questions on SO, after all. What do you all think?
Re: The Good and the Limitations of Github Copilot
#115> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…
> The author makes no comment on how hideously bad it is, which makes me suspect he didn’t notice, which… yeah, shows the problems of the whole thing. The author's comments on that "reverse" function are equally bad - https://youtu.be/9Pw-Roo_duE?t=171 It's described as "efficient" but it calls `len` on an unchanging list in 3 places.
Well, to be fair, the `len` operation on lists in Python is a constant time operation. What makes the example particularly bad is using a cast on the result of a floating-point division, rather than just using Python 3 integer division (i.e. the `//` operator). Copilot was clearly just spitting out Python 2 code here.
Re: The Good and the Limitations of Github Copilot
#116> Developers with experience can definitely handle this out but what if a newbie directly starts with the help of AI, he will spend more time on stack overflow than writing actual code. (oh wait that's how most of the developers are lmao) Spontaneous Ask HN: How much truth is there in this "devs be copy-pasting from SO all day" trope? Personally, I have used SO quite heavily in its early years, circa 2010-2014, inclu…
Based on my own experience, I remembered back in the day when I first started programming, in PHP, I copied many code from other commenters on PHP.net a lot (their online manual allows others to post code underneath the main docs). So I expect some new programmers to do the same but with StackOverflow.
Of course, I stopped doing that once I got hold on the real knowledge. In fact, now I look back the code that I copied from the others, it's just like watching a horror movie, and I rather rewrite the whole thing in my own term.
I guess the fairer statement to put in is "some programmer copy those code to 'get started'".
Re: The Good and the Limitations of Github Copilot
#117> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 , here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +.…
> The author makes no comment on how hideously bad it is, which makes me suspect he didn’t notice, which… yeah, shows the problems of the whole thing. The author's comments on that "reverse" function are equally bad - https://youtu.be/9Pw-Roo_duE?t=171 It's described as "efficient" but it calls `len` on an unchanging list in 3 places.
That’s actually roughly len(arr) times, not three, since two of the calls are inside the loop.
But the far bigger red flag there is that that it doesn’t just use arr.reverse(), which does the same thing and is typically 8–10× as fast in some simple testing (assuming a list), or arr[::-1], which makes a shallow copy rather than modifying the object in-place.
This matches what I’ve been seeing in code examples: Copilot likes to implement things from scratch rather than using libraries or even standard library functionality.
It’s possible that the word “array” tripped it up here and that it would have done something saner had it been told “list”, but I doubt it. (Python’s built-in array module is very seldom used; if you talk of arrays, you’re probably dealing with something like numpy’s arrays instead. But it’s far more likely that the built-in list type was what was desired here.)
There’s also one other significant point of bad and dangerous style in the code generated: the reverse function mutates its argument and returns it. Outside of fluent APIs (an uncommon pattern in Python, and not in use here), this is generally considered a bad idea in most languages, Python certainly included. It should either mutate its argument and return None, or not mutate its argument and return a new list.
Re: The Good and the Limitations of Github Copilot
#118Earlier quoted context omitted.
this is the worse take I've seen so far
Could you explain why or do you just want to feel superior? Both good choices honestly but I'd love to know your actual thoughts If I'm wrong tell me why and I'll happily reevaluate my opinion, promise :)
Well, my understanding is that your point is that it's not useful and only regurgitates code it already saw. That's super false, it's useful tool that generates code in a very context sensitive manner. Not always perfect, and in alpha, but still very useful.
The full copying only results from people actively probing the model to output copies of the code, and they made it work for very famous code that's been copied around github a bunch of times. That doesn't make it a non-useful piece of software.
Re: The Good and the Limitations of Github Copilot
#119Earlier quoted context omitted.
> The author makes no comment on how hideously bad it is, which makes me suspect he didn’t notice, which… yeah, shows the problems of the whole thing. The author's comments on that "reverse" function are equally bad - https://youtu.be/9Pw-Roo_duE?t=171 It's described as "efficient" but it calls `len` on an unchanging list in 3 places.
That’s actually roughly len(arr) times, not three, since two of the calls are inside the loop. But the far bigger red flag there is that that it doesn’t just use arr.reverse(), which does the same thing and is typically 8–10× as fast in some simple testing (assuming a list), or arr[::-1], which makes a shallow copy rather than modifying the object in-place. This matches what I’ve been seeing in code examples: Copilot…
> That’s actually roughly len(arr) times, not three, since two of the calls are inside the loop.
You're right! I noticed it a few minutes ago and changed the wording accordingly. Thanks for pointing out how shockingly bad that algorithm actually is! :D
> (...) the reverse function mutates its argument and returns it.
Yeah, that mutate + return is confusing. It's also worth noting that, as a result of the mutation, the function doesn't work on immutable types like strings and tuples.
Re: The Good and the Limitations of Github Copilot
#120Earlier quoted context omitted.
> The author makes no comment on how hideously bad it is, which makes me suspect he didn’t notice, which… yeah, shows the problems of the whole thing. The author's comments on that "reverse" function are equally bad - https://youtu.be/9Pw-Roo_duE?t=171 It's described as "efficient" but it calls `len` on an unchanging list in 3 places.
Well, to be fair, the `len` operation on lists in Python is a constant time operation. What makes the example particularly bad is using a cast on the result of a floating-point division, rather than just using Python 3 integer division (i.e. the `//` operator). Copilot was clearly just spitting out Python 2 code here.
I think the difference is that redundant re-calculation is a code smell in any language, at all times, whereas the float/int division issue requires knowledge of Python 2/3 syntax quirks.