Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI in software engineering at Google: Progress and the path ahead

research.google

Post reply on HN
Print view View original

171–180 of 245 posts

Re: AI in software engineering at Google: Progress and the path ahead

#171

Post by kibwen »

CoastalCoder wrote: post #150
discreteevent wrote: post #141

Earlier quoted context omitted.

> We coded in assembler, then used macros, then a language like C, Fortran, then more of Java/Go/Puthon/Rust/Ruby et al. The difference is that: 1) All of those things are deterministic [1] 2) In all of those cases I can debug at the level of the abstraction. [1] Meaning: Do exactly what I say. Don't make it up.

I agree with your overall point. In a certain sense I'd say optimizing compilers aren't deterministic: The same source code can produce different object code, depending on data and algorithms into which a typical programmer has little insight.

Crucially, optimizers are supposed to be semantics-preserving, and if they aren't that's considered a major problem.

Re: AI in software engineering at Google: Progress and the path ahead

#172

Post by calderwoodra »

williamcotton wrote: post #147
ks2048 wrote: post #15

Earlier quoted context omitted.

Seems they need to compare against "dumb" code completion. It seems that even when they are error-free, "large" AI-code-completions are just boilerplate that should be abstracted away in some functions rather than inserted into your code base. On a related note, maybe they should measure number of code characters that can be REMOVED by AI rather than inserted!

boilerplate that should be abstracted away in some functions rather than inserted into your code base Boilerplate is often tedious to write and just as often easy to read. Abstraction puts more cognitive load on the developer and sometimes this is not worth the impact on legibility.

Totally agree - LLMs remove the tedium of writing boilerplate code, which is often a better practice than abstracted code. But it takes years to experience to know what that's the case.

Re: AI in software engineering at Google: Progress and the path ahead

#173

Post by Rumudiez »

vineyardmike wrote: post #119
jprete wrote: post #6

Earlier quoted context omitted.

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

Frankly, its fine more often than we may care to admit. As the parent comment suggested, UI elements are a great candidate for this. Often very similar (how many apps have a menu bar, side bar, etc) and full of boilerplate. And at the rate things change on the front-end, it's often a candidate for frequent re-writes, so code quality and health don't need to be as strict. It'd be nice if every piece of software ever w…

UI is a terrible example to make your point. Tell me you don’t know frontend development…

Accessibility, cross browser+platform support, design systems, SEO, consistency and polish, you name it. You are most certainly not getting that from an LLM and most engineers don’t know how or don’t have a good eye for it to catch when the agent has gone astray or suggested a common mistake

Re: AI in software engineering at Google: Progress and the path ahead

#174

Post by zikzak »

kgilpin wrote: post #74

When AI is used properly, it’s doing one of two things: 1) Making non-controversial fixes that save time and take cognitive load off the developer. The best example is when code completion is working well. 2) It’s making you smarter and more knowledgeable by virtue of the suggestions it makes. You may discard them but you still learn something new, and having an assistant brainstorm for you enables a different mode o…

Not an AI expert but this sounds a lot like code review.

Most people are using systems to facilitate code review these days (we aren't sitting around boardrooms with printouts these days) so I wonder if there is a way to use the code review data streams combined with diffs to train AI to "do code reviews"?

Re: AI in software engineering at Google: Progress and the path ahead

#175

Post by jillesvangurp »

simonw wrote: post #165
jillesvangurp wrote: post #142

LLMs are undeniably useful for programming. The core challenge in making them more useful is the right UX for making this more seamless. I use intellij and things like codegpt. A few weeks ago they enabled auto complete. I disabled it after a day. Reason: most of the suggestions weren't great and it drowned out the traditional auto complete, which I depend on. I just found the whole thing too distracting. I also have…

> Instead I have to specify that I want it to generate a unit test, use kotlin-test and kotest-assertions, and not use backticks for the function names (doesn't work with kotlin-js). Every time. Have you experimented with GPTs much? I'd solve this problem by creating my own private "write a unit test" GPT that has my preferences configured in the system prompt.

I ask many things in a typical day. Creating custom GPTs for everything that pops in my head is not very practical. It's workaround for the fact that it doesn't remember anything about previous chats and conversations.

In an IDE context a lot of this stuff should be solved in the UI, not by the end user.

Re: AI in software engineering at Google: Progress and the path ahead

#176

Post by stuckinhell »

kgilpin wrote: post #74

When AI is used properly, it’s doing one of two things: 1) Making non-controversial fixes that save time and take cognitive load off the developer. The best example is when code completion is working well. 2) It’s making you smarter and more knowledgeable by virtue of the suggestions it makes. You may discard them but you still learn something new, and having an assistant brainstorm for you enables a different mode o…

"When AI is used properly" is a very very loaded statement. Especially since evidence shows code is one of the worst things its good at, it' a lot better at other tasks.

Re: AI in software engineering at Google: Progress and the path ahead

#177

Post by booleandilemma »

DebtDeflation wrote: post #151
bamboozled wrote: post #112

Earlier quoted context omitted.

So how long till AI will be fully replacing a SWE at Google? Notice parent said "fully"

It's the wrong question. It's like asking "How long until a bulldozer [1] fully replaces a human construction worker?" A bulldozer is not a full replacement for a human construction worker. However, 1 worker with a bulldozer can do the work of 10 workers with shovels. And 10 workers with bulldozers can build things no amount of workers with shovels can. [1] I'm using bulldozer as shorthand for all automated construct…

But then the 10 shovel-wielding humans are out of a job, right? The employer isn't just going to give everyone bulldozers.

It'll be one guy with a bulldozer doing the work of ten men.

Re: AI in software engineering at Google: Progress and the path ahead

#178

Post by chubot »

vineyardmike wrote: post #122
chubot wrote: post #91

Earlier quoted context omitted.

Yeah I kind of agree that when LLMs work REALLY well for autocomplete of your codebase -- that might be an indication that the language and library abstractions you use don't fit the problem very well. Code is read more than it's written. And it should be written to be read. If you are barfing out a lot of auto-completed stuff, it's probably not very easy to read. You have to read code to maintain it, modify it, anal…

> If you are barfing out a lot of auto-completed stuff, it's probably not very easy to read. From my experience using LLMs, I'd guess the opposite. LLMs aren't great at code-golf style, but they're great at the "statistically likely boilerplate". They max out at a few dozen lines at the extreme end, so you won't get much more than class structures or a method at a time, which is plenty for human-in-loop to guide it i…

Yes, it can make stuff that fits in the rest of the codebase

But I am saying it's not going to ever make the code significantly better

In my experience, code naturally gets worse over time as you add features, and make the codebase bigger. You have to apply some effort and ingenuity to keep it manageable.

So if everyone is using LLMs to barf out status quo code, eventually you will end up with something not very readable. It might look OK locally, but the readability you care about is a global concern.

Re: AI in software engineering at Google: Progress and the path ahead

#179

Post by mark_l_watson »

scarmig wrote: post #35
bigiain wrote: post #14

I'm looking forward to the day that some spicy autocomplete regurgitates an obvious chunk of AGPL code that it's stolen without permission or attribution - and it ends up in some critical part of Googles money printing machine, and the outside world finds out about it. I'm gonna need a _lot_ of popcorn.

I wouldn't be surprised if the sole training data for autocompletes was google3. It's an absolutely massive codebase, using the libraries and patterns Googlers use, and more or less entirely safe to train on. Any training data beyond that would be whitelisted by legal.

That was my assumption also. I worked at Google as a contractor in 2013 on a AI related project and their entire internal development environment was so much fun to use, and massive. Their internal coding world is an ideal and probably sufficient source of end-stage training data, given early training on the usual data sources to build language understanding first. They wouldn’t train on just their internal data exclusively.

Re: AI in software engineering at Google: Progress and the path ahead

#180

Post by pixiemaster »

stuckinhell wrote: post #176
kgilpin wrote: post #74

When AI is used properly, it’s doing one of two things: 1) Making non-controversial fixes that save time and take cognitive load off the developer. The best example is when code completion is working well. 2) It’s making you smarter and more knowledgeable by virtue of the suggestions it makes. You may discard them but you still learn something new, and having an assistant brainstorm for you enables a different mode o…

"When AI is used properly" is a very very loaded statement. Especially since evidence shows code is one of the worst things its good at, it' a lot better at other tasks.

every expert i talk to says: „LLMs are not good for my domain, but for $other_domain“
Post reply on HN