Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI in software engineering at Google: Progress and the path ahead

research.google

Post reply on HN
Print view View original

21–30 of 245 posts

Re: AI in software engineering at Google: Progress and the path ahead

#21

Post by jghn »

runlevel1 wrote: post #19
cplat wrote: post #4

Earlier quoted context omitted.

Yes. Although I’d like to see a deeper investigation. Of course, quality of completions have improved. But there could be a confounding phenomenon where newer folks might just be accepting a lot of suggestions without scrutiny.

I accept a lot of wrong suggestions because they look close enough that it's quicker for me to fix than it is to write the whole thing out from scratch. Which, IIUC, their metric captures: > Continued increase of the fraction of code created with AI assistance via code completion, defined as the number of accepted characters from AI-based suggestions divided by the sum of manually typed characters and accepted charac…

As someone who is generally pro-AI coding tools, I do the same. However sometimes it winds up taking me so much effort to fix the mistakes that it'd have been faster to do it myself. I often wonder if I'm seeing a net positive effort gain over time, I *think* so, but it's hard to measure.

Re: AI in software engineering at Google: Progress and the path ahead

#22

Post by danpalmer »

endisneigh wrote: post #7

Anyone who works at Google knows they are force feeding the employees with this stuff. Same thing they’re doing with the general public and search. Anyone who worked at google during + can see this is the same half hearted type of force feeding that will fail, with the rationalizations to boot. Disclaimer: I hold google stock but don’t work at google

I work at Google, this stuff is available, entirely optional, and in fact the most recent stuff is still something you have to sign up for and get on an opt-in basis in return for providing feedback/answering surveys.

Re: AI in software engineering at Google: Progress and the path ahead

#23

Post by endisneigh »

danpalmer wrote: post #22
endisneigh wrote: post #7

Anyone who works at Google knows they are force feeding the employees with this stuff. Same thing they’re doing with the general public and search. Anyone who worked at google during + can see this is the same half hearted type of force feeding that will fail, with the rationalizations to boot. Disclaimer: I hold google stock but don’t work at google

I work at Google, this stuff is available, entirely optional, and in fact the most recent stuff is still something you have to sign up for and get on an opt-in basis in return for providing feedback/answering surveys.

This is not true. You can’t even disable AI entirely in critique (AI suggested edits), not to mention cider.

Certain aspects can be disabled but you cannot entirely remove all AI affordances. It’s force feeding.

It will eventually all be opt-out. This is obvious to anyone who works at google. Again, same as force feeding of gen AI search

Re: AI in software engineering at Google: Progress and the path ahead

#24

Post by bongodongobob »

jprete wrote: post #6
trybackprop wrote: post #2

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets. In the past year since GPT-4 came out, I've also found this to be the case. I'm an ML/backend engineer with little experience in frontend development. Yet, I've…

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

Right because it's impossible for people to learn things themselves. Knowledge must only be passed down by The Gatekeeper.

I'm starting to feel like the programming community is just mad things are easier to learn now.

Re: AI in software engineering at Google: Progress and the path ahead

#25

Post by tmpz22 »

cloudking wrote: post #3

The most interesting chart is the "fraction of code created with AI assistance via code completion" trending up to 50%

Question for any Googlers in the thread - do folks speak up if they see flaws in the methodology or approach of this research or is the pressure from the top so strong on this initiative that people hush up?

Re: AI in software engineering at Google: Progress and the path ahead

#26

Post by imabotbeep2937 »

jjallen wrote: post #16
bigiain wrote: post #14

I'm looking forward to the day that some spicy autocomplete regurgitates an obvious chunk of AGPL code that it's stolen without permission or attribution - and it ends up in some critical part of Googles money printing machine, and the outside world finds out about it. I'm gonna need a _lot_ of popcorn.

How would the outside world find out about private Google code? Maybe the fact could be leaked? Unlikely though.

You're way, way under thinking the problem.

People who work at your bank right now are pasting your personal details into language models and asking if you deserve a loan. People will figure out how to get this data back out.

"But the model only uses old training data" there are myriad lawsuits in flight where these companies took information they shouldn't have, in all forms. Prompt engineers have already got the engines to spew things that are legally damning.

A real hack, which might archive user inputs as well as exfiltrate training data. We're only beginning to imagine the nightmare.

Can people use these language models to get private Google info? Only if Google was dumb enough to include that in the training. Hint: Yes, it's much, much worse than anyone imagines.

Re: AI in software engineering at Google: Progress and the path ahead

#27

Post by zackproser »

trybackprop wrote: post #2

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets. In the past year since GPT-4 came out, I've also found this to be the case. I'm an ML/backend engineer with little experience in frontend development. Yet, I've…

Having a similar experience but in the other direction. I have a lot of backend and frontend development experience but no ml background. Being able to ask stupid questions to get further faster has been making difference for me.

Re: AI in software engineering at Google: Progress and the path ahead

#28

Post by jprete »

refulgentis wrote: post #17
jprete wrote: post #6

Earlier quoted context omitted.

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

Source? This is unintuitive to me, I can't come up with a rationale.

I know of no review process that produces the same level of understanding as does authorship, because the author must build the model from scratch and so must see all the details, while the reviewer is able to do less work because they're fundamentally riding on the author's understanding.

In fact, in a high-trust system, e.g. a good engineering culture in a tech company, the reviewer will learn even less, because they won't be worried about the author making serious mistakes so they'll make less effort to understand.

I've experienced this from both sides of the transaction for code, scientific papers, and general discussion. There's no shortcut to the level of understanding given by synthesizing the ideas yourself.

Re: AI in software engineering at Google: Progress and the path ahead

#29

Post by imabotbeep2937 »

bongodongobob wrote: post #24
jprete wrote: post #6

Earlier quoted context omitted.

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

Right because it's impossible for people to learn things themselves. Knowledge must only be passed down by The Gatekeeper. I'm starting to feel like the programming community is just mad things are easier to learn now.

"Easier to learn" is abjectly false.

"Easier to get code submitted to a code base" is only marginally correlated with learning anything.

Re: AI in software engineering at Google: Progress and the path ahead

#30

Post by jstarks »

endisneigh wrote: post #23
danpalmer wrote: post #22

Earlier quoted context omitted.

I work at Google, this stuff is available, entirely optional, and in fact the most recent stuff is still something you have to sign up for and get on an opt-in basis in return for providing feedback/answering surveys.

This is not true. You can’t even disable AI entirely in critique (AI suggested edits), not to mention cider. Certain aspects can be disabled but you cannot entirely remove all AI affordances. It’s force feeding. It will eventually all be opt-out. This is obvious to anyone who works at google. Again, same as force feeding of gen AI search

Do you find this problematic?
Post reply on HN