Live data from Hacker News

Google CEO says more than a quarter of the company's new code is created by AI

businessinsider.com

291–300 of 1001 posts

Re: Google CEO says more than a quarter of the company's new code is created by AI

#291

To my experience, AIs can generate perfectly good code relatively easy things, the kind you might as well copy&paste from stackoverflow, and they'll very confidently generate subtly wrong code for anything that's non-trivial for an experienced programmer to write. How do people deal with this? I simply don't understand the value proposition. Does Google now have 25% subtly wrong code? Or do they have 25% trivial code…

> Does Google now have 25% subtly wrong code?

maybe the ai generates 100% of the company's new code, and then by the time the programmers have fixed it, only 25% is left of the AI's ship of Theseus

Re: Google CEO says more than a quarter of the company's new code is created by AI

#292
Without a clear explanation of methodology, this is meaningless. My guess is this statistic is generated using misleading techniques like classifying "code changes generated by existing bulk/automated refactoring tools" as "AI generated".

Re: Google CEO says more than a quarter of the company's new code is created by AI

#293
post #106

Earlier quoted context omitted.

> if this can be accompanied by an increase in software quality That’s a huge “if”, and by your own admission not what’s happening now. > other LLM agents reviewing code, feeding in compile errors, letting other LLM agents interact with the produced code, etc. What a stupid future. Machines which make errors being “corrected” by machines which make errors in a death spiral. An unbelievable waste of figurative and lit…

> What a stupid future. Machines which make errors being “corrected” by machines which make errors in a death spiral. An unbelievable waste of figurative and literal energy. Humans are machines which make errors. Somehow, we got to the moon. The suggestion that errors just mindlessly compound and that there is no way around it, is what's stupid.

> Humans are machines

Even if we accept the premise (seeing humans as machines is literally dehumanising and a favourite argument of those who exploit them), not all machines are created equal. Would you use a bicycle to fill your taxes?

> Somehow, we got to the moon

Quite hand wavey. We didn’t get to the Moon by reading a bunch of text from the era then probabilistically joining word fragments, passing that around the same funnel a bunch of times, then blindly doing what came out, that’s for sure.

> The suggestion that errors just mindlessly compound and that there is no way around it

Is one that you made up, as that was not my argument.

Re: Google CEO says more than a quarter of the company's new code is created by AI

#294

To my experience, AIs can generate perfectly good code relatively easy things, the kind you might as well copy&paste from stackoverflow, and they'll very confidently generate subtly wrong code for anything that's non-trivial for an experienced programmer to write. How do people deal with this? I simply don't understand the value proposition. Does Google now have 25% subtly wrong code? Or do they have 25% trivial code…

How would you react to a tech firm that in 2018, proudly announced that 25% of their code was generated by IntelliJ/Resharper/Visual Studio's codegen and autocomplete and refactoring tools?

Re: Google CEO says more than a quarter of the company's new code is created by AI

#295

Earlier quoted context omitted.

> the kind you might as well copy&paste from stackoverflow This bothers me. I completely understand the conversational aspect - "what approach might work for this?", "how could we reduce the crud in this function?" - it worked a lot for me last year when I tried learning C. But the vast majority of AI use that I see is...not that. It's just glorified, very expensive search. We are willing to burn far, far more fuel t…

we've decided we can't be bothered with traditional search Traditional search (at least on the web) is dying. The entire edifice is drowning under a rapidly rising tide of spam and scam sites. No one, including Google, knows what to do about it so we're punting on the whole project and hoping AI will swoop in like deus ex machina and save the day.

>> No one, including Google, knows what to do about it

I'm sure they can. But they have no incentive. Try to Google an item, and it will show you a perfect match of sponsored ads and some other not-so-relevant non-sponsored results

Re: Google CEO says more than a quarter of the company's new code is created by AI

#296

Earlier quoted context omitted.

we've decided we can't be bothered with traditional search Traditional search (at least on the web) is dying. The entire edifice is drowning under a rapidly rising tide of spam and scam sites. No one, including Google, knows what to do about it so we're punting on the whole project and hoping AI will swoop in like deus ex machina and save the day.

I personally think a big problem with search is major search engines try to be all things to all people and hence suffer as a result. For example: a beginner developer is possibly better served by some SEO-heavy tutorial blog post; an experienced developer would prefer results weighted towards the official docs, the project’s bug tracker and mailing list, etc. But since less technical and non-technical people vastly…

> But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later.

I mostly agree with your interesting comment, and I think your analysis basically jives with my sibling comment.

But one thing I take issue with is the idea that this type of thing is a good faith effort, because it’s more like a convenient excuse. Explaining substring search or even include/exclude ops to children and grandparents is actually easy. Setting preferences for tutorials vs API docs would also be easy. But companies don’t really want user-directed behavior as much as they want to herd users to preferred content with algorithms, then convince the user it was their idea or at least the result of relatively static ranking processes.

The push towards more fuzzy semantic search and “related content” everywhere is not to cater to novice users but to blur the line between paid advertisement and organic user-directed discovery.

No need to give megacorp the benefit of the doubt on stuff like this, or make the underlying problems seem harder than they are. All platforms land in this place by convergent evolution wherein the driving forces are money and influence, not insurmountable technical difficulties or good intentions for usability.

Re: Google CEO says more than a quarter of the company's new code is created by AI

#297
post #287

Earlier quoted context omitted.

Writing a prime-number factorization function is hardly "leetcode".

I didn't say it's hard, but it's most definitely leetcode, as in "pointless algorithmic exercise that will only show you if the candidate recently worked on a similar question". If that doesn't satisfy, here's a similar one at leetcode.com: https://leetcode.com/problems/distinct-prime-factors-of-prod... I would not expect a programmer of any seniority to churn stuff like that and have it working without testing.

> "pointless algorithmic exercise that will only show you if the candidate recently worked on a similar question".

I've been able to write one, not from memory but from first principles, any time in the last 40 years.

Re: Google CEO says more than a quarter of the company's new code is created by AI

#298
post #225

To my experience, AIs can generate perfectly good code relatively easy things, the kind you might as well copy&paste from stackoverflow, and they'll very confidently generate subtly wrong code for anything that's non-trivial for an experienced programmer to write. How do people deal with this? I simply don't understand the value proposition. Does Google now have 25% subtly wrong code? Or do they have 25% trivial code…

Most programming is trivial. Lots of non-trivial programming tasks can be broken down into pure, trivial sections. Then, the non-trivial part becomes knowing how the entire system fits together. I've been using LLMs for about a month now. It's a nice productivity gain. You do have to read generated code and understand it. Another useful strategy is pasting a buggy function and ask for revisions. I think most programm…

I would add that a lot of the time when I'm programming, I'm an expert on the problem domain but not the solution domain — that is, I know exactly what the pseudocode to solve my problem should look like; but I'm not necessarily fluent in the particular language and libraries/APIs I happen to have to use, in the particular codebase I'm working on, to operationalize that pseudocode.

LLMs are great at translating already-rigorously-thought-out pseudocode requirements, into a specific (non-esoteric) programming language, with calls to (popular) libraries/APIs of that language. They might make little mistakes — but so can human developers. If you're good at catching little mistakes, then this can still be faster!

For a concrete example of what I mean:

I hardly ever code in JavaScript; I'm mostly a backend developer. But sometimes I want to quickly fix a problem with our frontend that's preventing end-to-end testing; or I want to add a proof-of-concept frontend half to a new backend feature, to demonstrate to the frontend devs by example the way the frontend should be using the new API endpoint.

Now, I can sit down with a JS syntax + browser-DOM API cheat-sheet, and probably, eventually write correct code that doesn't accidentally e.g. incorrectly reject reject zero or empty strings because they're "false-y", or incorrectly interpolate the literal string "null" into a template string, or incorrectly try to call Element.setAttribute with a boolean true instead of an empty string (or any of JS's other thousand warts.) And I can do that because I have written some JS, and have been bitten by those things, just enough times now to recognize those JS code smells when I see them when reviewing code.

But just because I can recognize bad JS code, doesn't mean that I can instantly conjure to mind whole blocks of JS code that do everything right and avoid all those pitfalls. I know "the right way" exists, and I've probably even used it before, and I would know it if I saw it... but it's not "on the tip of my tongue" like it would be for languages I'm more familiar with. I'd probably need to look it up, or check-and-test in a REPL, or look at some other code in the codebase to verify how it's done.

With an LLM, though, I can just tell it the pseudocode (or equivalent code in a language I know better), get an initial attempt at the JS version of it out, immediately see whether it passes the "sniff test"; and if it doesn't, iterate just by pointing out my concerns in plain English — which will either result in code updated to solve the problem, or an explanation of why my concern isn't relevant. (Which, in the latter case, is a learning opportunity — but one to follow up in non-LLM sources.)

The product of this iteration process is basically the same JS code I would have written myself — the same code I wanted to write myself, but didn't remember exactly "how it went." But I didn't have to spend any time dredging my memory for "how it went." The LLM handled that part.

I would liken this to the difference between asking someone who knows anatomy but only ever does sculpture, to draw (rather than sculpt) someone's face; vs sitting the sculptor in front of a professional illustrator (who also knows anatomy), and having the sculptor describe the person's face to the illustrator in anatomical terms, with the sketch being iteratively improved through conversation and observation. The illustrator won't perfectly understand the requirements of the sculptor immediately — but the illustrator is still a lot more fluent in the medium than the sculptor is; and both parties have all the required knowledge of the domain (anatomy) to communicate efficiently about the sculptor's vision. So it still goes faster!

Re: Google CEO says more than a quarter of the company's new code is created by AI

#299
All this means is that 25% of code at Google is trivial boilerplate that would be better factored out of their process rather than tasking inefficient LLM tools with. The more they are willing to leave the “grunt work” to an LLM, the less likely they are to ever eliminate it from the process.

Re: Google CEO says more than a quarter of the company's new code is created by AI

#300
I think at this moment, this sounds more like "quarter of the company's new code is created using stackoverflow and other forums. Many many people use all these tools to find information, as they did using stackoverflow a month ago, but now suddenly we can call it "created by AI". It'd be nice to have a distinction. I'm saying this, while being very excited about using LLMs as a developer.
Post reply on HN