Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI in software engineering at Google: Progress and the path ahead

research.google

Post reply on HN
Print view View original

51–60 of 245 posts

Re: AI in software engineering at Google: Progress and the path ahead

#51

Post by zackproser »

mensetmanusman wrote: post #44
skywhopper wrote: post #41

Earlier quoted context omitted.

I’m not seeing any evidence that any of this is actually good for Google’s business. Their observations that half of code checked in is from suggestions by an LLM is not really surprising in a regimented dev platform with tons of boilerplate. That stat tells us nothing about actual code quality, development velocity, or skill curves over time, much less business impact. What product of Google’s has been improved by t…

Google prints money until search goes away. Nothing else they work on has to succeed.

But increasingly I prefer to ask LLMs the same things I used to search Google for...

Re: AI in software engineering at Google: Progress and the path ahead

#52

Post by userbinator »

irrational wrote: post #32
bongodongobob wrote: post #24

Earlier quoted context omitted.

Right because it's impossible for people to learn things themselves. Knowledge must only be passed down by The Gatekeeper. I'm starting to feel like the programming community is just mad things are easier to learn now.

Are they learning? Or are they just accepting the results, as long as it appears to more or less do what they want, and moving on? The danger seems to be code that is syntactically correct and compiles without errors, but is logically incorrect.

Or more insidiously, is correct for everything except the edge cases.

Re: AI in software engineering at Google: Progress and the path ahead

#53

Post by yeahwhatever10 »

refulgentis wrote: post #18
endisneigh wrote: post #7

Anyone who works at Google knows they are force feeding the employees with this stuff. Same thing they’re doing with the general public and search. Anyone who worked at google during + can see this is the same half hearted type of force feeding that will fail, with the rationalizations to boot. Disclaimer: I hold google stock but don’t work at google

That's not true, I'm a xoogler as of October, and at least 2 of my ex-colleagues continue to generally wonder if AI can write code or not, and if it can, they haven't tried it. Last update 60 days ago. It does look like there's an auto-installed cider extension, which is fine, the worst case for this stuff is "it's in my autocomplete list" -- that's fine!

You guys really got knock it off with adding “oogler” to everything.

Re: AI in software engineering at Google: Progress and the path ahead

#54

Post by trybackprop »

jprete wrote: post #6
trybackprop wrote: post #2

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets. In the past year since GPT-4 came out, I've also found this to be the case. I'm an ML/backend engineer with little experience in frontend development. Yet, I've…

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

[dead]

Re: AI in software engineering at Google: Progress and the path ahead

#55

Post by userbinator »

As others have mentioned, unless you have a strong conscience and really know what you're doing, it's far too tempting to just accept AI-generated suggestions without really thinking, and IMHO losing that understanding is a dangerous path to go down. AI can only increase quantity, not quality. The industry desperately needs far more of the latter.

Related: https://navendu.me/posts/ai-generated-spam-prs/

Re: AI in software engineering at Google: Progress and the path ahead

#56

Post by javawizard »

skywhopper wrote: post #41
zackproser wrote: post #34

After reading this I'm wondering how the indy code autocomplete tools are going to be able to compete longterm with this giant feedback rich data machine Google has built...do engineering orgs of sufficient scale ultimately hoard their tooling for competitive advantage, thereby leaving independent players to cater to developers outside of Google? Feels like yes...but plenty of inventions trickle out in various forms.

I’m not seeing any evidence that any of this is actually good for Google’s business. Their observations that half of code checked in is from suggestions by an LLM is not really surprising in a regimented dev platform with tons of boilerplate. That stat tells us nothing about actual code quality, development velocity, or skill curves over time, much less business impact. What product of Google’s has been improved by t…

Xoogler here.

This exactly. There is so much boilerplate involved in writing anything inside Google.

AI was great to cut that down a bit. It's still nowhere near what it's like in the outside and/or non-Java world.

Which isn't to say that this isn't progress - just that that stat should be taken with context.

Re: AI in software engineering at Google: Progress and the path ahead

#57

Post by bamboozled »

userbinator wrote: post #55

As others have mentioned, unless you have a strong conscience and really know what you're doing, it's far too tempting to just accept AI-generated suggestions without really thinking, and IMHO losing that understanding is a dangerous path to go down. AI can only increase quantity, not quality. The industry desperately needs far more of the latter. Related: https://navendu.me/posts/ai-generated-spam-prs/

It's a concern I have too, when I get tired, I start to just delegate to co-pilot suggestions as I get desperate, if I didn't have co-pilot, I'd probably just log off for the day.

I actually don't really use copilot as I didn't find it that helpful, so I don't really have the problem anymore, but I could see it was a danger. Bit like driving when tired.

Re: AI in software engineering at Google: Progress and the path ahead

#58

Post by notpachet »

desolved wrote: post #13
bdjsiqoocwk wrote: post #11

Earlier quoted context omitted.

Also, people shouldn't be allowed to use computers unless they understand how transistors work. If you don't have the depth of knowledge you get nothing.

It took me way too long to realize this was a joke.

It's HN so you can't be entirely sure.

Re: AI in software engineering at Google: Progress and the path ahead

#59

Post by gbnwl »

gerdesj wrote: post #39
runlevel1 wrote: post #12

Humans have limited RAM, so we have to put our ideas into an external medium that can then be refined. I've been finding AI's suggestions -- even when rather wrong -- help me do that initial step faster. Which, I think, jives with their findings here.

"Humans have limited RAM" I would suggest we have flexible RAM. Also, we have an awful lot of it. The analogy breaks down as soon as you look at it too seriously! In IT we largely deal with compute, persistent storage and non-persistent storage. Roughly speaking: CPU, RAM, HDD. In humans we might be considered to have similar "abilities" but unlike IT there is a mostly a single thing that performs all of those functi…

Just because the brain is a single “thing” doesn’t mean it doesn’t have distinct types of memory. Consider looking up “working memory” as it’s probably the best analogue to RAM here.

Re: AI in software engineering at Google: Progress and the path ahead

#60

Post by vitus »

From the screencast:

> implement also for Days

This fails to recognize that this is a bad feature that the Abseil library would explicitly reject (hence the existence of absl::CivilDay) [0], and instead perpetuates the oversimplification that 1 day is exactly 24 hours (which breaks at least twice every year due to DST).

Which is to say: it'll tell you how to do the thing you ask it to do, but will not tell you that it's a bad idea.

And, of course, that assumes that it even makes the change correctly in the first place (which is nowhere near guaranteed, in my experience). I have seen (and bug-reported!) cases where it incorrectly inverts conditionals, introduces inefficient or outright unsafe code, causes unintended side effects, perpetuates legacy (discouraged) patterns, and more.

It turns out that ML-generated code is only as good as its training data, and a lot of google3 does not adhere to current best practices (in part due to new library developments and adoption of new language versions, but there are also many corners of the codebase with, um, looser standards for code quality).

[0] https://github.com/abseil/abseil-cpp/blob/bde089f/absl/time/...

Post reply on HN