This is a tacit admission it was Mythos, not Gemini, despite alluding to experiments with Gemini earlier on:
> Added support for model interoperability to leverage the unique strengths of both open-weights and proprietary models
Google fixed more Chrome bugs in June than over the past two years, thanks to AI
271–280 of 668 posts
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#272Earlier quoted context omitted.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
Just like humans, AI needs a "reality" to check things against. If we don't provide that in an automated way, somehow, it will always make mistakes and we'd have to point it out. Since, at that point, we're just mediating with what reality gives us and what an AI should do with it. AI doesn't have enough senses yet. It's trapped in a box.
[deleted]
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#273The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#274Earlier quoted context omitted.
Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you? It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
This right here is the truth. Too often people talk in abstract ways, not disclosing what they're actually working on. And then they tell their experiences as universal facts.
The reason they don’t disclose what they’re working on is because it’s always some CRUD app
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#275Earlier quoted context omitted.
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
> Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! Tests onli validate the presence of bugs, not their abscence (Djikstra). I'll also add that tests look at outputs and don't care how those outputs are derived. E.g. code filtering the entire db in memory will be fine in tests.
Haven't you ever written tests that assert that certain things happen or do not happen along the way? We use mocks all the time to, say, make sure that the code doesn't load the entire db in memory. Then it's there as a regression if somebody down the line wants to do that exact thing for some reason.
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#276I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
I’ve had a lot of success with this. Give it a good enough harness, leave it for a few days and see what it finds that actually holds up.
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#277Earlier quoted context omitted.
But them it's more like AI assisted performance fuzzing? Just throw stuff against some measuring harness and see what sticks?
Yes, sort of. Not that there is anything wrong with that. Fuzzing yields tons of trash results, the occasional fleck of gold and the even more rare nugget. Then you analyze how it found the flakes and nuggets update your fuzzing harness. Every so often you get to write a paper or give a talk about a breakthrough to share meaningful improvements. Current AI isn't super effective at making the breakthroughs, but it sur…
What does that mean? Democratising the breakthroughs you can point LLMs at?
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#278This is only a flex if AI also didn’t cause an increase in bugs that needed to be fixed.
So, your conjecture is that the LLM's skill to detect bugs somehow magically disappears the moment they start writing new code?
No, the extra code velocity allowed by AI without proper QA support means there could have been a lot more bugs introduced into products over the last year or so, giving AI more bugs to find. It’s tongue in cheek but not completely uncalled for. It’s been long known that one dishonest way to inflate your bug fix count is to put more bugs into the product to fix. Goodhart’s law is merciless.
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#279Earlier quoted context omitted.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker. 2. Code reviews and security reviews happen quicker and produce more findings. I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues. Even the Linux Kernel (I'd say Windows and Apple too)…
Linus: "it keeps finding embarrassing bugs" Linux Kernel: https://lore.kernel.org/all/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80Kzs... The idea that software has gotten so complex that a machine can evaluate code paths better than a human, seems to bristle the fur of many. Some people didn't think we would see the day where that comparative human limitation was laid bare in simpler tasks than they expected. I believe older develo…
I don't know, this more and more feels like a sentiment projected on to people than anything real these days.. Like even in that linked thread, is personal offense like you lay out here really were you can place Laurent Pinchart's push back? You don't read anything else there at all?
And either way, what, we are going to keep this line going for another 5 years? Aren't you bored?
Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI
#280A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing. I think I'll abandon this discussion and keep using AI qu…
Just exploit the edge while it lasts lol