We're going to invest $7 trillion to get... better autocomplete? Apple and Microsoft are gonna smoke these guys.
Where did you get 7 trillion?
AI in software engineering at Google: Progress and the path ahead
161–170 of 245 posts
Re: AI in software engineering at Google: Progress and the path ahead
#162When AI is used properly, it’s doing one of two things: 1) Making non-controversial fixes that save time and take cognitive load off the developer. The best example is when code completion is working well. 2) It’s making you smarter and more knowledgeable by virtue of the suggestions it makes. You may discard them but you still learn something new, and having an assistant brainstorm for you enables a different mode o…
> So how can AI usefully suggest design-level and conceptual ideas in a way that doesn’t require a user “trigger”? Within the IDE, I’m not sure.
https://upload.wikimedia.org/wikipedia/en/d/db/Clippy-letter...
Re: AI in software engineering at Google: Progress and the path ahead
#163Earlier quoted context omitted.
Are they learning? Or are they just accepting the results, as long as it appears to more or less do what they want, and moving on? The danger seems to be code that is syntactically correct and compiles without errors, but is logically incorrect.
One data point: In our startup we are short on frontend software engineers. Our project manager started helping with the UI using an IDE (cursor a VS-code fork) with native ChatGPT integration. In the span of six months, they have become very proficient at React. They had wanted to learn basic frontend coding for multiple years but never managed to pass the initial hurdles. Initially, they were only accepting suggest…
With the number of anti patterns in React, are you sure everything’s OK? The thing’s about declarative is that it’s more like writing equations. Imperative have a tighter feedback loop, and ultimately only code organization is a issue. But so many things can go wrong with declarative as it’s one level higher in the abstraction stack.
I’m not saying that React is hard to learn. But I believe buying a good book would have them get there quicker.
Re: AI in software engineering at Google: Progress and the path ahead
#164The most interesting chart is the "fraction of code created with AI assistance via code completion" trending up to 50%
Question for any Googlers in the thread - do folks speak up if they see flaws in the methodology or approach of this research or is the pressure from the top so strong on this initiative that people hush up?
There were already teams building ML-based code completion, code suggestions, and code repair before LLMs blew up a couple years ago. So the principle of it isn't driven by AI hype.
Yes, there are oodles of people complaining about AI overuse and there is a massive diversity of opinion about these tools being used for coding, testing, LSCs, etc. I've seen every opinion from "this is absolute garbage" to "this is utter magic" and everything in between. I personally think that the AI suggestions in code review are pretty uniformly awful and a lot of people disable that feature. The team that owns the feature tracks metrics on disabling rates. I also have found the AI code completion while actually writing code to be pretty good.
I also think that the "% of characters written by AI" is a pretty bad metric to chase (and I'm stunned it is so high). Plenty of people, including fairly senior people, have expressed concern with this metric. I also know that relevant teams are tracking other stuff like rollback rates to establish metrics around quality.
There is definitely pressure to use AI as much as reasonably possible and I think that at the VP and SVP level it is getting unreasonable, but at the director and below level I've found that people are largely reasonable about where to deploy AI, where to experiment with AI, and where to AI to fuck off.
Re: AI in software engineering at Google: Progress and the path ahead
#165LLMs are undeniably useful for programming. The core challenge in making them more useful is the right UX for making this more seamless. I use intellij and things like codegpt. A few weeks ago they enabled auto complete. I disabled it after a day. Reason: most of the suggestions weren't great and it drowned out the traditional auto complete, which I depend on. I just found the whole thing too distracting. I also have…
> Instead I have to specify that I want it to generate a unit test, use kotlin-test and kotest-assertions, and not use backticks for the function names (doesn't work with kotlin-js). Every time.
Have you experimented with GPTs much? I'd solve this problem by creating my own private "write a unit test" GPT that has my preferences configured in the system prompt.
Re: AI in software engineering at Google: Progress and the path ahead
#166Earlier quoted context omitted.
Are they learning? Or are they just accepting the results, as long as it appears to more or less do what they want, and moving on? The danger seems to be code that is syntactically correct and compiles without errors, but is logically incorrect.
Oh come on, the user is talking about building UIs. I don't know how else you learn. Your attitude just reeks of high-horse. As if it was better to learn things from stackoverflow.
Who learned stuff from stack overflow? In my own case, it was all books, plus a few videos. Stack Overflow was mostly to know why things has gone (errors not explicit enough) or very specific patterns. And there was a peer review system which lent credibility to answers.
Re: AI in software engineering at Google: Progress and the path ahead
#167When AI is used properly, it’s doing one of two things: 1) Making non-controversial fixes that save time and take cognitive load off the developer. The best example is when code completion is working well. 2) It’s making you smarter and more knowledgeable by virtue of the suggestions it makes. You may discard them but you still learn something new, and having an assistant brainstorm for you enables a different mode o…
> So how can AI usefully suggest design-level and conceptual ideas in a way that doesn’t require a user “trigger”? Within the IDE, I’m not sure. https://upload.wikimedia.org/wikipedia/en/d/db/Clippy-letter...
Clippy was on the right path at the wrong time, the issue wasn't the concept of an assistant that watches your work and provides help, the issue was it could detect a letter, offer help but the help it gave you was just a bunch of shallow formatting suggestions without context to the actual work.
An actual assistant that can preempt what you need and create it before you get there with a 95% success rate will not feel like Clippy.
Re: AI in software engineering at Google: Progress and the path ahead
#168Earlier quoted context omitted.
I agree with your overall point. In a certain sense I'd say optimizing compilers aren't deterministic: The same source code can produce different object code, depending on data and algorithms into which a typical programmer has little insight.
Being a bit pedantic. Globally but not locally deterministic
Your brain's not deterministic.
Re: AI in software engineering at Google: Progress and the path ahead
#169LLMs are undeniably useful for programming. The core challenge in making them more useful is the right UX for making this more seamless. I use intellij and things like codegpt. A few weeks ago they enabled auto complete. I disabled it after a day. Reason: most of the suggestions weren't great and it drowned out the traditional auto complete, which I depend on. I just found the whole thing too distracting. I also have…
> Instead I have to specify that I want it to generate a unit test, use kotlin-test and kotest-assertions, and not use backticks for the function names (doesn't work with kotlin-js). Every time. Have you experimented with GPTs much? I'd solve this problem by creating my own private "write a unit test" GPT that has my preferences configured in the system prompt.
This, as well as system prompts could help you a bunch.
Re: AI in software engineering at Google: Progress and the path ahead
#170Earlier quoted context omitted.
"Humans have limited RAM" I would suggest we have flexible RAM. Also, we have an awful lot of it. The analogy breaks down as soon as you look at it too seriously! In IT we largely deal with compute, persistent storage and non-persistent storage. Roughly speaking: CPU, RAM, HDD. In humans we might be considered to have similar "abilities" but unlike IT there is a mostly a single thing that performs all of those functi…
These kinds of similies make less and less sense nowadays because we've got nvme storage nowadays, and that can be as fast as 7GByte/s. That's a lot faster then the RAM in most devices today. And with less latency too. RAMs differentiating factor is increasingly just that it can handle a lot of read/write cycles, not it's speed. And that doesn't map to anything in biology
Wat? NVMe storage is much slower than RAM. And orders of magnitude slower than VRAM.