[Warning: long comment ahead]
I haven't used Copilot yet (I'm not coding in any of the languages it covers)
but I've tried to understand it from the few articles (most critical) posted
about it on the internet and from my understanding of program synthesis and
languae modelling. I don't fully trust lay users to be able to exercise a
machine learning model comprehensively enough to precisely understand its pros
and cons, but I think I've formed some intuition about how Copilot works from
the demonstrations I've seen. Here's my thoughts:
a) As TFA points out, Copilot was trained on _all_ code on github (for select
languages). "All" includes good code and bad code, code with bugs and code with,
er, fewer bugs, code that does what it should and code that doesn't. The result
is that Copilot models _all_ that code, regardless of whether it's got bugs or
not.
b) Copilot is (based on) a language model that predicts the next sequence of
tokens, not a discriminator of good/bad code. That means it can generate code
but it can't generate "correct" code. It doesn't have a concept of
correct/incorrect code.
b.1) More generally, it's impossible to discriminate between "correct" and
"incorrect" code from features (tokens, in the case of language models) alone.
The correctness of code depends on the programer's intent. In program synthesis,
the programmer's intent is encoded as a specification and the correctness of a
synthesised program must be considered in the context of that specification. A
language model could be trained to include a specification as a feature, or as
an additional modality (similar to training on images and text together to
generate captions for images, or images from captions). However, this is not how
Copilot was trained. The upshot of all this is that Copilot can't generate
"correct programs" because it doesn't know what "correct programs" are.
b.2) But wait! Copilot can generate code from a specification! From a natural
language specification, in the form of inline comments given as part of a
prompt. This seems to be a favourite mode of testing Copilot and its results are
impressive, but there are two problems: i) natural language specifications are
imprecise and get more so the longer they grow; and, ii) Copilot still has no
way to determine whether the code it generates satisfies a natural language
specification, or not.
b.3) For program synthesis with guarantees of correctness, a specification is
not enough: a verification procedure is also necessary -and those tend to be
very tricky because Halting Problem.
c) Copilot has memorised at least part of its training set. This is evident in
the demonstrations of verbatim generation of known code, e.g. the Inverse Square
Root function (from Quake... III Arena I think?) etc.
c.1) This observation is helpful in understanding language models in general.
The assumption is often stated that a model can't possibly be memorising its
training set because models are smaller on disk than their training sets. And
yet it seems that large language models are memorising parts of their
training sets- most likely, the ones that are more common, and therefore most
likely.
c.2) This in turn means that the programs that Copilot is most likely to generate
are variations of programs included in its training set and that have been
memorised. Why is this most likely? Because those programs are themselves "most
likely" - they are programs with the highest prior probability in the space of
programs represented by Copilot's model.
c.3.1) That said, Copilot doesn't always generate memorised programs verbatim. It
clearly has the ability to "mix and match" and generate programs that are not
included in its training set. How does Copilot (or language models in general)
achieve that? A language model is essentially a representation of a dense region
of Cartesian space bounded by the elements (tokens, expressions, whole programs)
of the training set. Anything within that dense geometric region, a language
model can recognise and generate. Anything outside it, is down to luck.
c.3.2) To generate programs not included in its training set, Copilot
interpolates between the programs in its training set and other programs in
its model ("interpolates" as in "finds a point on a gradient between two
programs' representations"). However, this interpolation is stochastic.
Fudging very, very much, we can imagine Copilot traversing the AST of a
memorised program and applying slight peturbations to that AST, at random,
though according to its learned prior probabilities. This is why Copilot will
generate different code each time it's run even with the same prompt, and the
generated code is closer or farther away from memorised code according to its
"temperature" parameter (that controls the randomness of "perturbations").
c.4) Suppose you take some program P and walk down its AST, perturbing the nodes
at random, without having any way to know whether a perturbation results in
correct code, incorrect code, or pure garbage. What is the expected result? The
expected result is correct code, incorrect code, or garbage. Remember: Copilot
can't tell which is which. So we don't know which we'll get.
d) All that said, I think that i) Copilot can make a great boilerplate generator
but ii) only if people learn to use it right. So far I think most demonstrations
I've seen are "holding it wrong".
d.1) As to ii) above, I think the best use-case of Copilot is to generate short
code snippets, spanning at most a couple of lines, rather than entire functions.
I think it's also going to work a lot better if it's allowed to complete code
rather than comments (with code). See point b.2 above. The demonstrations I've
seen so far seem to try to get Copilot to act as an Autopilot (sorry, couldn't
resist). In that, it fails in the ways it should be expected to fail . I suspect
that as users get more familiar with Copilot's abilities the best way to use it
will become evident and Copilot will take its rightful place as a useful tool in
the programmer's toolbox that can simplify, but not replace, day-to-day
programming work.
e) Rumours of the imminent demise of the programming profession have, of course,
been greately exaggerated. Wait another couple of generations, maybe. Human
generations.
f) If you're interested in program synthesis and the progress that has been
achieved so far in this old, but still active field of research, the following
is a good recent survey of the field:
https://www.microsoft.com/en-us/research/wp-content/uploads/...