Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google fixed more Chrome bugs in June than over the past two years, thanks to AI

blog.google

Post reply on HN
Print view View original

421–430 of 668 posts

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#421

Post by rpdillon »

iLoveOncall wrote: post #38
dabedee wrote: post #16

How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.

> The post has counts for everything that went right and nothing for what could go wrong. That's AI for you. At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments. No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.

This is kind of a weird take.

If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures? The general way technology is deployed is that we try to find ways to make it work, because those are the most interesting. We're not as interested in all the ways it doesn't work.

From my perspective, some people are trying to use AI in the same way somebody might use a laptop to paddle a canoe. Sure, you can do it, but it's not a good idea. The fact that it doesn't work well is not particularly interesting.

If I steelman your position, I guess the ideal repository would be a set of cases where it's known to work well and a set of cases where it's known to not work well. A little bit like ProtonDB or SteamDB, perhaps.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#422

Post by dingaling »

rpdillon wrote: post #416
hn_submit wrote: post #389

To me this merely signals how broken C++ development really is. Most if not all of the bugs being uncovered are memory related and therefore intimately tied to the mental memory model of C and C++, namely manual memory management. It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability. C and C++ are simply not fit for purpose when large scale software projects are concer…

The entire compute world was built on top of C and C++. Saying they're not fit for purpose doesn't seem correct. I think what you're really saying is they're not an optimal choice from a safety perspective. > All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem. Seems like hyperbole... It might be ideal if we could snap our fingers and suddenly have rust ports, but I feel like…

> Saying they're not fit for purpose doesn't seem correct.

I'll quite happily say they were never fit for purpose. C had a role in bootstrapping early software development in the absence of anything better but once Lisp, Delphi, Smalltalk and Java arrived on the scene then it, and its equally footgunnable variants, should have been pushed to the margins where hardware constraint was the primary consideration.

Writing a browser in C++ in the 21st century was ridiculous and irresponsible.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#423

Post by rpdillon »

goldenarm wrote: post #69
Cthulhu_ wrote: post #58

Earlier quoted context omitted.

Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example. But it's an open source project, you can go and figure out whether your assertion is correct.

The single 13yo issue is anecdata. They obviously have the full git blame statistics but chose not to include them in the blog post, which is a bit concerning.

It's weird to assume that this is all smoke and mirrors where AI is fixing AI's bugs. When we dove into the rsync issue, what was happening is that AI was revealing bugs that had been previously unknown, resulting in the need to fix them. I would be stunned if that were not the case here.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#424

Post by zahlman »

buzzin__ wrote: post #145
goldenarm wrote: post #51

Elephant in the room : how many of these bugs were written by LLMs in the first place ? Because creating 100x more bugs and fixing 100x more isn't something to be proud of.

If you assume that AI can produce code with less bugs, your 100 times increase in fresh bugs means you also increased your speed for new functionality by more than 100 times. If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.

> If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.

As if humans are incapable of spotting critical bugs simply because they've existed for a long time? And we know humans don't produce fresh bug-free code, because the old bugs exist.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#425

Post by artrockalter »

In my experience LLMs do find a lot of embarrassing bugs as Linus says but it can constantly turn into a game of whack-a-mole where most of the bugs it finds were written in previous LLM sessions. It's a huge struggle to get it to actually fix the root cause of the bug instead of patching the symptom.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#426

Post by skydhash »

user43928 wrote: post #392
cautiouscat wrote: post #382

Earlier quoted context omitted.

If it’s 4 months of hard work, then it’s not like you’re blindly doing it as the top commenter mentioned.

That is true, I was mostly speaking about a supposed need to review the code. My effort instead goes to manual QA testing, providing feedback and preferences, and asking questions. The limitation here is mostly that the model does not know what looks and feels good. With improvements to the vision capabilities and a better understanding of motion, or what looks appealing to humans, the implementation could probably h…

> That is true, I was mostly speaking about a supposed need to review the code.

Until you go to prod, you can believe a lot of things about the state of your code. Reviewing code is not merely about “Does this things work”. Testing and linting do cover most of that. Reviewing is mostly about: Will this design cover the current set of constraints (some may be conformance) and can it evolve? Are the assumptions correct? Is the security layer good enough?…

The user interface part of the code is only one single component out of many.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#427

Post by hn_submit »

rpdillon wrote: post #416
hn_submit wrote: post #389

To me this merely signals how broken C++ development really is. Most if not all of the bugs being uncovered are memory related and therefore intimately tied to the mental memory model of C and C++, namely manual memory management. It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability. C and C++ are simply not fit for purpose when large scale software projects are concer…

The entire compute world was built on top of C and C++. Saying they're not fit for purpose doesn't seem correct. I think what you're really saying is they're not an optimal choice from a safety perspective. > All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem. Seems like hyperbole... It might be ideal if we could snap our fingers and suddenly have rust ports, but I feel like…

I assure you that Microsoft knew for years that there were thousands if not tens of thousands of memory related bugs lurking in Windows. But their policy was "don't fix" (and "don't look") unless a remotely exploitable vulnerability was making headlines. Because every minute a developer spends finding and fixing bugs is a minute he can't spend on adding new features customers will pay for.

Luckily one of the perks of A.I. is that it makes large scale code migrations feasible in a timely manner.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#428

Post by mccr8 »

truncate wrote: post #5

Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.

AI is very good at finding security bugs. If you are working on a product that processes hostile data like a web browser and has a security bug bounty program, your choices are either to find them yourself or be overwhelmed by external submissions.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#429

Post by fn-mote »

lilbigdoot wrote: post #378
sirsinsalot wrote: post #336

Earlier quoted context omitted.

Even if true, this kind of response is unhelpful and fosters dismissal of your point. This kind of attitude more broadly paints AI advocates as cultists because they refuse to engage beyond "you're doing it wrong". How about give pointers on "doing it right?"

Exactly. I'm a huge FP fan. When someone doesn't like FP I don't lecture them on how they're holding it wrong. I usually assume that, unsurprisingly, the tool doesn't suit every problem or there may be a design issue in the tool they ran into

> I usually assume that […] the tool doesn't suit every problem

My expectation: inexperienced FP is trying to shoehorn a mutation-based approach into a framework that performs best with immutable data, or is manually recreating familiar structure (for loop) instead of using the idiomatic way.

But what do you tell someone in this case? They really need to learn more about the basics. Which is how I am interpreting the GGP’s dismissive-sounding comment.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#430

Post by jdm2212 »

gtowey wrote: post #414
hn_submit wrote: post #389

To me this merely signals how broken C++ development really is. Most if not all of the bugs being uncovered are memory related and therefore intimately tied to the mental memory model of C and C++, namely manual memory management. It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability. C and C++ are simply not fit for purpose when large scale software projects are concer…

I'll await your Rust ports of Windows, Linux and MacOS then. This isn't really a fair take. Keep in mind that C/C++ has been the backbone of the most important software in the world since the 1970s. At that time we didn't have virtually unlimited compute and memory at our fingertips the way we do now. It was a huge improvement to have a high level language which still could be optimized nearly as well as assembly. It…

They're working on it for Linux: https://rust-for-linux.com/
Post reply on HN