Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

simonwillison.net

Post reply on HN
Print view View original

21–30 of 93 posts

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#21

Post by PunchyHamster »

baq wrote: post #8
dreadnip wrote: post #3

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely will find issues. If I keep running the same prompt, it will keep finding issues. I’ve spammed GitHub PR reviews and it just keep finding (or inventing?) new issues. There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model wi…

You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.

I've had it find bug, I asked it to make test to trigger the bug, and then it figured out it's not a bug. It will absolutely do wish fulfilment

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#23

Post by memoriyato3 »

MallocVoidstar wrote: post #13
baq wrote: post #8

Earlier quoted context omitted.

You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.

No, depending on the complexity of the issue models can be into loops, where they go "this is definitely an issue and must be fixed", and then the resulting fixed code gets "this is definitely an issue and must be fixed", and then the resulting fixed code has the original 'issue'.

yeah, happened to me: "A is very wrong, you should do B", and on the next fresh review loop "B is very wrong, you should do A"

typically this means there is some ambiguity in the specification, and the model flips between alternative interpretations

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#24

Post by TripolitianFish »

Myzel394 wrote: post #20
dreadnip wrote: post #3

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely will find issues. If I keep running the same prompt, it will keep finding issues. I’ve spammed GitHub PR reviews and it just keep finding (or inventing?) new issues. There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model wi…

That's just plain wrong. The new models do not hallucinate as much as they used to (in my personal experience)

> plain wrong > (in my experience) What are you even saying.

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#25

Post by bfjvibybd6cuvu6 »

MallocVoidstar wrote: post #13
baq wrote: post #8

Earlier quoted context omitted.

You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.

No, depending on the complexity of the issue models can be into loops, where they go "this is definitely an issue and must be fixed", and then the resulting fixed code gets "this is definitely an issue and must be fixed", and then the resulting fixed code has the original 'issue'.

That's a different kind of loop.

For a normal review loops you can ask the model to return with nothing found if nothing is found and not invent things and it will do a better job of exiting without anything found.

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#26

Post by imhoguy »

dreadnip wrote: post #3

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely will find issues. If I keep running the same prompt, it will keep finding issues. I’ve spammed GitHub PR reviews and it just keep finding (or inventing?) new issues. There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model wi…

You need to create review skill and there define what "issue" or "good" are for you to limit sensitiviness. Otherwise you depend on model's random threshold or non of such then you get perfection chasing.

Anyway it will never match your judgemend completely unless you upload your brain dump into model.

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#27

Post by left-struck »

PunchyHamster wrote: post #21
baq wrote: post #8

Earlier quoted context omitted.

You didn’t do it enough. They stop finding bugs eventually. Also, different models can find different bugs (though they do find the same ones, too, which is good and expected). For best results you want to run multi model reviews in loops. If you had multiple people look at your PRs multiple times on different days results would be very similar.

I've had it find bug, I asked it to make test to trigger the bug, and then it figured out it's not a bug. It will absolutely do wish fulfilment

Yeah when these models find a bug i like to ask it to write a test that will fail if the bug is real and pass when the bug is solved.

It’s not perfect but usually it works pretty well, and I’ve had the model come back to me with oh actually the test passed, the bug doesn’t work exist

As a bonus, you’ve now got a test that can detect that bug if it comes up again.

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#28

Post by memoriyato3 »

embedding-shape wrote: post #12
dreadnip wrote: post #3

The problem I have with this workflow is that the models are still too eager to please. If I ask it to scan a release and note possible issues, it absolutely will find issues. If I keep running the same prompt, it will keep finding issues. I’ve spammed GitHub PR reviews and it just keep finding (or inventing?) new issues. There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model wi…

> There is never a “Nothing found, good to go!”. Like when you do recursive programming, have you tried providing more/better stop conditions? If you literally just say "Continue until there are no more issues" then it'll do just that, but if you scope it better, like "Only mention issues related to X, Y or that leads to Z" and so on, you'll get less noise and more focus on issues that actually matter (to you).

also helps adding negative conditions like "do not nitpick", or specific bad attractors that you see "do not investigate/report anything related to symlinks, they are not a concern"

Re: sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

#30

Post by mirekrusin »

5701652400 wrote: post #22

just a note. in most parts of the world 149.25 USD can cover utilities, water, and food for a month for 1 adult person or even a family.

In others it's pizza night for family or half a bill for sushi dinner, so what?
Post reply on HN