Claude 4 • Hacker News Board

Re: Claude 4

#741

Post by BeetleB » Fri, May 23, 2025, 2:08 AM UTC

fumeux_fume wrote: post #672

BeetleB wrote: post #531

Earlier quoted context omitted.

I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to ha…

Aside from the fact that you seem to be demanding a lot from someone who's informally sharing their experience online, I think the effectiveness really depends on what you're writing code for. With straightforward use cases that have ample documented examples, you can generally expect decent or even excellent results. However, the more novel the task or the more esoteric the software library, the likelier you are to…

> Aside from the fact that you seem to be demanding a lot from someone who's informally sharing their experience online

Looking at isolated comments, you are right. My point was that it was a trend. I don't expect everyone to go into details, but I notice almost none do.

Even what you pointed out ("great for somethings, crappy for others") has much higher entropy.

Consider this, if every C++ related submission had comments that said the equivalent of "After using C++ for a few weeks, my verdict is that its performance capabilities are unimpressive", and then didn't go into any details about what made them think that, I think you'd find my analogous criticism about such comments fair.

Re: Claude 4

#742

Post by AstroBen » Fri, May 23, 2025, 2:14 AM UTC

siavosh wrote: post #509

briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

You'd think there should be some sort of standard "morality/ethics" pre-prompt for all of these.

at this point that would be like putting a leash on a chair to stop it running away

Re: Claude 4

#743

Post by dbingham » Fri, May 23, 2025, 2:14 AM UTC

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase.

If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut it. The hallucinations are still a serious issue, and even when they aren't hallucinating they do not generate quality code. Their code is riddled with bugs, bad architectures, and poor decisions.

Writing good code with an LLM isn't any faster than writing good code without it, since the vast majority of an engineer's time isn't spent writing -- it's spent reading and thinking. You have to spend more or less the same amount of time with the LLM understanding the code, thinking about the problems, and verifying its work (and then reprompting or redoing its work) as you would just writing it yourself from the beginning (most of the time).

Which means that all these companies that are firing workers and demanding their remaining employees use LLMs to increase their productivity and throughput are going to find themselves in a few years with spaghettified, bug-riddled codebases that no one understands. And competitors who _didn't_ jump on the AI bandwagon, but instead kept grinding with a strong focus on quality will eat their lunches.

Of course, there could be an unforeseen new order of magnitude jump. There's always the chance of that and then my prediction would be invalid. But so far, what I see is a fast approaching plateau.

Re: Claude 4

#744

Post by BeetleB » Fri, May 23, 2025, 2:16 AM UTC

osn9363739 wrote: post #727

sponnath wrote: post #578

Earlier quoted context omitted.

I feel like the opposite is true but maybe the issue is that we both live in separate bubbles. Often times I see people on X and elsewhere making wild claims about the capabilities of AI and rarely do they link to the actual output. That said, I agree that AI has been amazing for fairly closed ended problems like writing a basic script or even writing scaffolding for tests (it's about 90% effective at producing tests…

I'm in the same bubble. I find if they do link to it it's some basic unimpressive demo app. That said, I want to see a video where of one of these people that apparently 10x'd there programming go against a dev without AI across various scenarios. I just think it would be interesting to watch if they had a similar base skill & understanding of things.

> That said, I want to see a video where of one of these people that apparently 10x'd there programming go against a dev without AI across various scenarios.

It would be interesting, but do understand that if AI coding is totally fantastic in one domain (basic automation scripting) and totally crappy in another (existing, complex codebase), it's still a (significant) improvement from the pre-AI days.

Concrete example: A few days ago I had an AI model write me a basic MCP tool: Creating a Jira story. In 15 minutes, it had written the API function for me, I manually wrapped it to make it an MCP tool, tested it, and then created tens of stories from a predefined list, and verified it worked.

Now if you already know the Jira APIs (endpoints, auth, etc), you could do it with similar speed. But I didn't. Just finding the docs, etc would take me longer.

Code quality is fine. This is not production code. It's just for me.

Yes, there are other Jira MCP libraries already. It was quicker for me to write my own than to figure out the existing ones (ditto for Github MCP). When using LLMs to solve a coding problem is faster than using Google/SO/official docs/existing libraries, that's clearly a win.

Would I do it this way for production code? No. Does that mean it's bad? No.

Re: Claude 4

#745

Post by -__---____-ZXyw » Fri, May 23, 2025, 2:17 AM UTC

gmueckl wrote: post #620

GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

Isn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?

It might be the ultimate irony if we were training them. But we aren't, at least not in the sense that we train dogs. Dogs learn, and exhibit some form of intelligence. LLMs do not.

It's one of many unfortunate anthropomorphic buzz words which conveniently wins hearts and minds (of investors) over to this notion that we're tickling the gods, rather than the more mundane fact that we're training tools for synthesising and summarising very, very large data sets.

Re: Claude 4

#746

Post by kweingar » Fri, May 23, 2025, 2:20 AM UTC

jjmarr wrote: post #568

GolDDranks wrote: post #437

After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of tha…

As a junior developer it's much easier for me to jump into a new codebase or language and make an impact. I just shipped a new error message in LLVM because Cline found the 5 spots in 10k+ files where I needed to make the code changes. When I started an internship last year, it took me weeks to learn my way around my team's relatively smaller codebase. I consider this a skill and cost issue. If you are rich and able…

> If you cannot afford the hundreds of dollars a month Claude costs

Employers will buy AI tools for their employees, this isn't a problem.

If you're saying that you need to buy and learn these tools yourself in order to get a job, I strongly disagree. Prompting is not exactly rocket science, and with every generation of models it gets easier. Soon you'll be able to pick it up in a few hours. It's not a differentiator.

Re: Claude 4

#747

Post by mayukh » Fri, May 23, 2025, 2:20 AM UTC

troad wrote: post #731

GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

> I think that people tend to forget what LLMs really are. [...] They do not have a plan, they do not have thoughts of their own. > A LLM is smart enough to [...] I thought this was an interesting juxtaposition. I think we humans just naturally anthropomorphise everything, and even when we know not to, we do anyway. Your analysis is correct, I think. The reason we find this behaviour frightening is because it appears…

Beautifully written. Interestingly, humans also don't know definitively where their own thoughts arise from

Re: Claude 4

#748

Post by SamBam » Fri, May 23, 2025, 2:22 AM UTC

mellow_observer wrote: post #560

SamBam wrote: post #545

This is the first LLM that has been able to answer my logic puzzle on the first try without several minutes of extended reasoning. > A man wants to cross a river, and he has a cabbage, a goat, a wolf and a lion. If he leaves the goat alone with the cabbage, the goat will eat it. If he leaves the wolf with the goat, the wolf will eat it. And if he leaves the lion with either the wolf or the goat, the lion will eat the…

Actual answer: He crosses the river and takes all of the animals and the cabbage with him in one go. why not? Alternative Answer: He just crosses the river. Why would he care who eats what? Another Alternative Answer: He actually can't cross the river since he doesn't have a boat and neither the cabbage nor the animals serve as appropriate floatation aids

Yup, for your first one, no LLM has ever realized that I don't actually specify that he can only take one thing at a time. I think that's natural that it would assume that (as would most humans) because it would be so heavily primed to fill that in from every other version of the puzzle it's seen.

Re: Claude 4

#749

Post by mrheosuper » Fri, May 23, 2025, 2:24 AM UTC

jjmarr wrote: post #568

GolDDranks wrote: post #437

After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of tha…

As a junior developer it's much easier for me to jump into a new codebase or language and make an impact. I just shipped a new error message in LLVM because Cline found the 5 spots in 10k+ files where I needed to make the code changes. When I started an internship last year, it took me weeks to learn my way around my team's relatively smaller codebase. I consider this a skill and cost issue. If you are rich and able…

Some companies do not like you upload their code to 3rd parties

Re: Claude 4

#750

Post by SamBam » Fri, May 23, 2025, 2:24 AM UTC

albumen wrote: post #586

KolmogorovComp wrote: post #559

Earlier quoted context omitted.

Had you paid more attention, you would have realised it's not the classic riddle, but an already tweaked version that makes it impossible to solve, hence why it is interesting.

Mellowobserver above offers three valid answers, unless your puzzle also clarified that he wants to get all the items/animals across to the other side alive/intact.

Indeed, but, no LLM has ever realized that I don't actually specify that he can only take one thing at a time. It's natural that it would assume that (as would most humans) because it would be so heavily primed to fill that in from every other version of the puzzle it's seen.

I'd give them full credit if they noticed that, but I was also wanting to see if, given the unstated assumptions (one thing in the boat, don't let anything eat anything else, etc) they'd realize it was unsolvable.