Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google fixed more Chrome bugs in June than over the past two years, thanks to AI

blog.google

Post reply on HN
Print view View original

381–390 of 667 posts

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#381

Post by QuercusMax »

swat535 wrote: post #358
Tade0 wrote: post #86

Earlier quoted context omitted.

It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

Additionally, if you are asking agents to write test, be very careful. They will write tautology tests. They will mock things to no end. They will flat out REMOVE assertions (saying it's not needed). They can also write test to assert the wrong result. You have to always review it, it's exhausting.

This was my experience with some of the stupider models, but I haven't had Claude do this type of thing since I started using it for work about 6mo ago.

Claude will do some boneheaded things for sure, but it's pretty good about writing tests that are useful, and not removing or modifying tests just because they're in the way.

Claude is pretty bad about assuming that it couldn't have broken a test it didn't know about, as it has often told me "this is already broken on main" which is definitely NOT true.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#382

Post by cautiouscat »

user43928 wrote: post #330
skydhash wrote: post #300

Earlier quoted context omitted.

Unless the app is released and running on user’s device, your “it works” is on the level of hackathon’s demo. Being on prod has always been the true testing ground of code. It seems like when people are talking about production level quality and how the AI. should be helping them there, plenty of people comes up with their “it works on my machine” anecdotes.

I use the app every day and I am confident in the quality. I will launch it by the end of August. After ~4 months of hard work.

If it’s 4 months of hard work, then it’s not like you’re blindly doing it as the top commenter mentioned.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#383

Post by ghm2199 »

Code related Vulnerability discovery is one of those bright spots where AI can shine because its shaped as a learning-test: every example it runs gives it feedback that is deterministic. If you have ever pointed claude/codex to your a binary executable and asked it to figure out something it can run against full throttle, I fond it always gets back to you with some good insights.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#384

Post by redsocksfan45 »

sirsinsalot wrote: post #336
bmurphy1976 wrote: post #294

Earlier quoted context omitted.

Let me translate this for you: "I have not spent the time cultivating the soft skills necessary to leverage this tool successfully therefore it's the tool that sucks."

Even if true, this kind of response is unhelpful and fosters dismissal of your point. This kind of attitude more broadly paints AI advocates as cultists because they refuse to engage beyond "you're doing it wrong". How about give pointers on "doing it right?"

[dead]

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#385

Post by qsort »

bonoboTP wrote: post #357
qsort wrote: post #66

Earlier quoted context omitted.

Very hard to say anything definitive on this because it's a moving target, but last time I tried models still had a distinct sense of "consistently good, sometimes great at micro, bad at macro". Similar to how, even for relatively pedestrian CRUD, they'll do code that's objectively fine at the function/file/class level but can still make a mess if you don't supervise them at least at a high-level.

It is moving so fast that your experience with older models is irrelevant today. Sorry. Claude Code, Fable 5, xhigh reasoning, allow it to run the full CI, end to end and benchmark, it will not make silly mistakes (or only occasionally). Also, be able to state what you desire. Have any docs or materials in the same directory so the model can reference it. For even better results: turn on speech recognition and braind…

I'm not saying I don't get any use out of them, fwiw. In a lot of ways they write perfectly fine code, I'm just saying without some guidance at the macro-level they lose the forest for the trees, that's all.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#386

Post by lilbigdoot »

stack_framer wrote: post #301
VBprogrammer wrote: post #17

I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…

The useless suggestions are exhausting. One of my co-workers uses Claude for all asynchronous communication, including Slack messages, Jira comments, code reviews, and emails. Every single message from him, literally every time he communicates, it's a massive wall of text, overflowing with scope-creep suggestions like nothing I've ever seen before. It's impossible to ask him a simple question and get a simple answer.…

I had a model yesterday suggest that I intern all float literals in my compiler to save on memory lmao

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#387

Post by bmurphy1976 »

sirsinsalot wrote: post #336
bmurphy1976 wrote: post #294

Earlier quoted context omitted.

Let me translate this for you: "I have not spent the time cultivating the soft skills necessary to leverage this tool successfully therefore it's the tool that sucks."

Even if true, this kind of response is unhelpful and fosters dismissal of your point. This kind of attitude more broadly paints AI advocates as cultists because they refuse to engage beyond "you're doing it wrong". How about give pointers on "doing it right?"

My comment is dismissive because the parent comment is also dismissive. But you are right, advice is helpful. It's in my comment, but it may not be obvious to everybody, so here it is:

Work on your project management, communication, and mentorship skills. The AI tools are much more like a team of sloppy but capable junior engineers and not a reliably consistent fabrication machine. Change your perception that one simple prompt is going to magically solve a hard problem the first time. Learn to break projects down, organize them into sub-projects, and implement them in steps. Learn to communicate more clearly, define your requirements more rigorously, and adapt around your "team's" strengths and weaknesses.

We want these things to be rigorous and flawless but reality is messy just like working with a regular team of engineers. They are good at some things, terrible at others, constantly changing, and unreliable. But we've historically built tons of reliable software on unreliable actors by focusing on process, collaboration, and communication.

AKA, not just the hard technical skills, but the soft skills that allow unreliable software teams to thrive. They don't know what they don't know and it's your job to manage that.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#388

Post by ndriscoll »

maccard wrote: post #49
glimshe wrote: post #41

A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing. I think I'll abandon this discussion and keep using AI qu…

> A lot of people here seem to be living in a different universe than me I feel this way on this topic too. > I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other…

I've found that these models are extremely good at reasoning since late last year, but they're basically 200 IQ Amelia Bedelias. So my working model earlier this year was to give it a brief instruction and watch how it failed, which was generally a lack of context that it couldn't have possibly known. Then I'd add that context to the team AI docs repo, try again from scratch, and iterate until it did what I wanted without me having to give a bunch of detail in the prompt. Over the course of 2-3 months, I needed to give new context and steering less and less.

I also encouraged my team to paste any inquiries or alerts we got in our slack channel verbatim into codex and follow that procedure until it could figure it out as a one-shot.

I still treat it as more of a design and coding partner day-to-day, so I'm typically not trying to one-shot, but it can e.g. be triggered by a pagerduty alert, reference our code, check grafana panels, query application state across different servers/clusters, and come up with bug fix PRs all autonomously, and then post its analysis, mitigation suggestions, and PR link to slack for the on-caller to review. This was all just some docs, scripts, and a little listener service to trigger a one-shot model prompt that we built.

As a design partner, it's made it easier to add more telemetry or create rapid prototypes to check my assumptions. I've encouraged my team to regularly ponder on what facts they wish they knew, no matter how difficult it would be to find out. Pretend you could just ask God anything you want to know. Then ask the AI to go add whatever's needed to find out. We should all move toward a perfect intuition for what our system is doing because all questions can now be easily answered if we can only think to ask them.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#389

Post by hn_submit »

To me this merely signals how broken C++ development really is. Most if not all of the bugs being uncovered are memory related and therefore intimately tied to the mental memory model of C and C++, namely manual memory management.

It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability.

C and C++ are simply not fit for purpose when large scale software projects are concerned. All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem.

The hundreds if not thousands of developers working on Chrome weren't idiots who didn't know what they're doing. The complexity of programming in C/C++ is simply beyond most intelligent individuals' ability to get perfect all the time.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#390

Post by lilbigdoot »

Supermancho wrote: post #250
nevi-me wrote: post #21

Earlier quoted context omitted.

1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker. 2. Code reviews and security reviews happen quicker and produce more findings. I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues. Even the Linux Kernel (I'd say Windows and Apple too)…

Linus: "it keeps finding embarrassing bugs" Linux Kernel: https://lore.kernel.org/all/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80Kzs... The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older develo…

I don't know why people keep acting like any kind of LLM skepticism or criticism is based on bristled fur or something.

I am very skeptical of these workflows, but I also use LLMs regularly. I specifically use them because they are better than me at sifting through massive amounts of complex information. They are also quite, uh, sketchy, for things that are significantly easier. A total mixed bag in my experience that ultimately is useful, and I will continue to use

Post reply on HN