I've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.
The trick to coding with LLMs is not caring if the code is correct.
Making o1, o3, and Sonnet 3.7 hallucinate for everyone
191–200 of 233 posts
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#192Earlier quoted context omitted.
What tools do you use to perform such tasks?
Aider. I tried Cursor too, but I don't like VS Code and not being able to chose the LLM provider. I think there are already a lot of tools that perform kind of equally.
What do you mean? You can chose the LLM in Cursor. (I don't like VSCode either, but unfortunately Cursor is best for the rest of the UX; I find myself using Cursor for the prompt-based part of the work, and then move to IntelliJ to do "my" part of the coding)
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#193I've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.
I've used it for some smaller greenfield code with success. Like, write an Arduino program that performs a number of super-sampled analog readings, and performs a linear regression fit, printing the result to the serial port. That sort of stuff can be very helpful to newbies in the DIY electronics world for example. But for anything involving my $dayjob it's been fairly useless beyond writing unit test outlines.
> That sort of stuff can be very helpful to newbies
Yep, if it's newbie stuff there are enough tutorials out there for the LLM to have data.
It's when you get off the tutorials and do the actual functionality of your project that they fail.
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#194Earlier quoted context omitted.
I would love an AI coding assistant that doesn't respond until it's gone out and tested its answer in a sandbox to confirm it actually compiles.
I personally haven't figured out why there isn't a tool that just loops the AI severals times on a task by compiling, feeding in the errors adjusting and repeating and then letting the user review the result be in a success or failure by exceed the loop limits.
[dead]
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#195Earlier quoted context omitted.
That's not just small, it's utterly miniscule. It's most certainly not large.
Nah this is miniscule: https://github.com/coreutils/coreutils/blob/master/src/yes.c You can fit a hell of a lot of functionality in 3k statements. Really whether it's considered large or small necessarily must rely on the functionality it's intended to provide.
You're thinking on "lean" vs "bloated". "Large" and "small" have meanings of their own, and a 3k LOC project wouldn't be accepted as "large" by anyone. "Small", maybe.
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#196Earlier quoted context omitted.
I would love an AI coding assistant that doesn't respond until it's gone out and tested its answer in a sandbox to confirm it actually compiles.
I personally haven't figured out why there isn't a tool that just loops the AI severals times on a task by compiling, feeding in the errors adjusting and repeating and then letting the user review the result be in a success or failure by exceed the loop limits.
Claude has a tendency to overcomplicate bug fixes. If you put it on loop it would probably build a custom ERP platform.
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#197ChatGPT used to assure me that you can use JS dot notation to access elements in a Python dict. It also invented Redocly CLI flags that don't exist. Claude sometimes invents OpenAPI specification rules. Any time I ask anything remotely niche, LLMs are often bad.
I once asked Perplexity (using Claude underneath) about some library functionality, which it totally fabricated. First, I asked it to show me a link to where it got that suggestion, and it scolded me saying that asking for a source is problematic and I must be trying to discredit it. Then after I responded to that it just said “this is what I thought a solution would look like because I couldn’t find what you were as…
How is Perplexity even able to give invalid results? Isn't it parsing the web first then drawing a conclusion?
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#198I've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.
The trick to coding with LLMs is not caring if the code is correct.
Call me cynical but coding at some companies has such perverse incentives that I kind of get it:
- "Solve" the issue assigned to me with a bunch of code that looks about right. Passes review and probably not covered by tests anyway.
- Once QA or customers notice it's not working, I can get credit for "solving" the bug as well.
- Repeat for 0 value delivered but infinite productivity points in my next performance review.
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#199Earlier quoted context omitted.
GPT can't even tell what its done or give what it knows it should. It's endless, "Apologies, here is what you actually asked for ..." and again it isn't.
I would love an AI coding assistant that doesn't respond until it's gone out and tested its answer in a sandbox to confirm it actually compiles.
Devin does that
Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone
#200Earlier quoted context omitted.
I would love an AI coding assistant that doesn't respond until it's gone out and tested its answer in a sandbox to confirm it actually compiles.
Devin does that
Wasn't Devin a scam? Honest question; Google is not very clear about it but some people said it was a scam/money grab. You are saying it works?