Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"ChatGPT said this" Is Lazy

terriblesoftware.org

Post reply on HN
Print view View original

141–150 of 161 posts

Re: "ChatGPT said this" Is Lazy

#141

Post by deaux »

xigoi wrote: post #138
deaux wrote: post #129

Earlier quoted context omitted.

> I do not use AI for engineering work and never will > Once someone outsources their brain they are unlikely to keep learning or evolving from that point It doesn't piss me off, it makes me feel sorry for you. Sorry that you're incapable of imagining those with curiosity who use AI to do exactly what you're claiming they don't - learning. The uncurious were uncurious before AI and remain so. They never asked why, th…

> User: Question. LLM: Answer. User: Why A? Why B? LLM: Answer. User: But C. What's the tradeoff? LLM: Answer. User: Couldn't also X? LLM: Answer. User: I'm not familiar with Y, explain. In practice it’s more like: User: Question. LLM: Answer. User: Why A? LLM: Actually, you’re right, it’s not A, it’s B. User: Why B? LLM: Actually, you’re right, it’s not B, it’s C. User: Why C? LLM: Actually, you’re right, it’s not C…

Change your question asking strategy, and potentially models. GPT-5 is much less prone to do this. That's the easiest change to make, try it out. I'm not saying this to promote OpenAI, have a look at my profile if you must. Qwen is also less likely to do this. Sonnet and GPT4o are obviously famous for this sycophancy, and DeepSeek has clearly been training on Claude outputs, and so is also prone to it.

Of course never use chat models, only straight API, as the system prompts for consumer versions make them induce sycophancy as that is what the uncurious like. For this conversation, don't use things like Claude Code or other harnesses either. Just have a conversation through API calls.

You also changed the questions in the scenario in a way that makes them more likely. Keep them open-ended, the curious' most used phrase should probably be "Why or why not? If so, explain why. If not, explain why not".

Re: "ChatGPT said this" Is Lazy

#142

Post by lrvick »

simonw wrote: post #120
lrvick wrote: post #113

Earlier quoted context omitted.

I do not doubt that LLMs might some day be able to generate something like my work in stagex, but it would only be because someone trained one on my work and that of other people that insist on solving new problems by hand. Even then, I would never use it, because it would be some proprietary model I have to pay some corpo for either with my privacy, my money, or both... and they could take it away at any time. I do…

> Even then, I would never use it, because it would be some proprietary model I have to pay some corpo for either with my privacy, my money, or both... and they could take it away at any time. Have you been following the developments in open source / open weight models you can run on your own hardware? They're getting pretty good now, especially the ones coming out of China. The GLM, Qwen and DeepSeek models out of C…

Not only have I been following a lot of the open models, you may find it surprising I have extensively tested some of them and coerced them to generate deterministic responses across different machines as a method to prove responses are not tampered with, as well as developing ways to run them in remotely attestable secure enclaves to ensure people that use them for sensitive applications can have provable privacy with end to end encryption.

I will admit that I find deploying and hacking on the tech itself super interesting. Hell I founded a machine learning company and got a paper published with the AAAI for my cheap bulk training data acquisition techniques back in 2012 before most cared about this stuff.

I even think there are a ton of great and exciting use cases for this tech. Like identifying cancer in large photographic datasets, etc. I have a lot of hope about medical applications in particular.

All that said, I just don't think LLMs are remotely competitive or useful at the type of threat modeling, security engineering, and auditing work I do on average. They are the wrong tool for my job, which require a level of actual reasoning that LLMs are nowhere near capable of right now, or are likely to be any time soon. Maybe they could help with a script here and there which might save me a few hours a month, but for 95%+ of it, they would just waste my time regurgitating the same industry standard bad advice and approaches that I am trying to change while making me duller at writing code by hand when I need to.

As contrast though, I would not fire someone for using LLMs for learning or inspiration as long as they consistently prove they fully understand and can explain every line of every PR they submit, can pair program or usefully contribute to engineering discussions without LLMs, and maintain a competitive level of quality with the rest of the team. Not everyone has to make the same tool choices I do, as long as they can hold their own in a team with me and are not dumb enough to regurgitate AI slop they don't understand.

It is amusing you use vegans as an example. I am not a vegan, but I often describe myself as something of a digital vegan that is very very selective about what tools I use and what I expect from them such as why I also don't use a smartphone or GPS.

Re: "ChatGPT said this" Is Lazy

#143

Post by deaux »

kalleboo wrote: post #135
deaux wrote: post #129

Earlier quoted context omitted.

> I do not use AI for engineering work and never will > Once someone outsources their brain they are unlikely to keep learning or evolving from that point It doesn't piss me off, it makes me feel sorry for you. Sorry that you're incapable of imagining those with curiosity who use AI to do exactly what you're claiming they don't - learning. The uncurious were uncurious before AI and remain so. They never asked why, th…

The problem with current LLMs is that they are so sycophantic that they don't tell you when you're asking the wrong questions. I've been down the path of researching something with an LLM, and I came to a conclusion. I later revisited the subject and this time I instead read a longer-form authoritative source and it was clear that "interrogating" the matter meant I missed important information.

Which have you been using and through which interface? Try using GPT-5 through the API, without a harness. Whatever model you use, the key phrase is "Why or why not? If so, explain why. If not, explain why not.".

Telling when you're asking the wrong questions is also a skill that the curious such as yourself will develop in no time through experience. Are you saying this didn't give a shred of intuition how to do so in the future?

Re: "ChatGPT said this" Is Lazy

#144

Post by lmf4lol »

insin wrote: post #26

I'm starting to run into the other end of this as a reviewer, and I hate it. Stories full of nonsensical, clearly LLM-generated acceptance requirements containing implementation details which are completely unrelated to how the feature actually needs to work in our product. Fine, I didn't need them anyway. PRs with those useless, uniformly-formatted LLM-generated descriptions which don't do what a PR description shou…

Stupid question maybe, but are there no company guidelines? How can this be acceptable in the company culture?

Re: "ChatGPT said this" Is Lazy

#145

Post by yeasku »

deaux wrote: post #129
lrvick wrote: post #23

I do not use AI for engineering work and never will, because doing the work of thinking for myself is how I maintain the neural capacity for forming my own original thoughts and ideas no AI has seen before. If anyone gives me an opinion from an AI, they disrespect me and themselves to a point they are dead to me in an engineering capacity. Once someone outsources their brain they are unlikely to keep learning or evol…

> I do not use AI for engineering work and never will > Once someone outsources their brain they are unlikely to keep learning or evolving from that point It doesn't piss me off, it makes me feel sorry for you. Sorry that you're incapable of imagining those with curiosity who use AI to do exactly what you're claiming they don't - learning. The uncurious were uncurious before AI and remain so. They never asked why, th…

You feel sorry for us?

Re: "ChatGPT said this" Is Lazy

#147

Post by ninkendo »

tensility wrote: post #98
ninkendo wrote: post #37

Earlier quoted context omitted.

Eh… your complaint describes every single piece of information available on the internet. Let’s try it with other stuff: “Looking at solutions on stack overflow outsources your brain” “Searching arxiv for literature on a subject outsources your brain” “Reading a tutorial on something outsources your brain” There’s nothing that makes ChatGPT et al appreciably different from the above, other than the tendency to halluc…

That tendency to hallucinate that you so conveniently downplay is a major problem. I'll take reading the reference manual myself all day rather than sifting through the output of a bullshit generator.

I'm not trying to downplay it, it absolutely is a major problem.

But it's a very different problem from "outsourcing your brain". In fact, the tendency to hallucinate is the very reason you still need to use your brain when using an LLM.

As I said, it's not any different from a web search or a wikipedia read: It's one input, and you still have to think for yourself.

Re: "ChatGPT said this" Is Lazy

#148

Post by LocalH »

Leherenn wrote: post #10
einsteinx2 wrote: post #3

I’ve noticed this trend in comments across the internet. Someone will ask or say something, the someone else will reply with “I asked ChatGPT and it says…” or “According to AI…” ChatGPT is free and available to everyone, and so are a dozen other LLMs. If the person making the comment wanted to know what ChatGPT had to say, they could just ask it themselves. I guess people feel like they’re being helpful, but I just d…

Isn't it the modern equivalent of "let me Google that for you"? My experience is that the vast majority of people do 0 research (AI assisted or not) before asking questions online. Questions that could have usually been answered in a few seconds if they had tried. If someone preface a question by saying they've done their research but would like validation, then yes it's in incredibly poor taste.

The unholy blend of that is when someone "asks Google" and regurgitates the AI summary.

udm=14 my beloved

Re: "ChatGPT said this" Is Lazy

#149

Post by D-Machine »

lrvick wrote: post #139
D-Machine wrote: post #126

Earlier quoted context omitted.

It is increasingly incredibly important here to make a distinction between using an LLM with and without search. Without search, I agree with you. But e.g. ChatGPT with search enabled is often an invaluable research tool, and dramatically speeds up finding relevant sources. It basically automates the spidering of references and links, and also handles the basic checks for semantic relevance quite well, and this task…

When someone is able to demonstrate they are more efficient than me in my areas of research and expertise by utilizing an LLM, I will at least pay attention to their workflow and investigate ways to replicate it with FOSS tools I can host, improve, debug, and control for myself. Until then, LLMs are of no interest to me. I am pretty good at using grep on local data caches.

Yup, reasonable enough. In areas where you are a true expert (i.e. where there are only handfuls at most of people like you, or where all expert knowledge is not online and accessible by LLMs), you essentially already know where to look, and how to most effectively phrase searches, so a generic LLM that automatically crawls common databases and sites is going to be useless, almost by definition / necessity. Training is just fitting a manifold to training data (interpolating and extrapolating), but truly expert domains (as defined earlier) don't generally have enough data for these things to learn helpful semantics.

Perhaps a model fine-tuned or specifically built for the niche domains might help non-experts and people learning to better do semantic / common-language searches, and this might accelerate learning. But, it might not. Not all domains have their core knowledge encoded in documents and/or language, and summarization is sometimes only harmful (especially for highly technical writing).

It also seems to me almost definitional though that LLMs aren't going to beat true expertise, because LLMs aren't true intelligence or GAI, they really do just estimate token path densities / curve-fit. And real understanding is clearly more than that.

Re: "ChatGPT said this" Is Lazy

#150

Post by simonw »

lrvick wrote: post #142
simonw wrote: post #120

Earlier quoted context omitted.

> Even then, I would never use it, because it would be some proprietary model I have to pay some corpo for either with my privacy, my money, or both... and they could take it away at any time. Have you been following the developments in open source / open weight models you can run on your own hardware? They're getting pretty good now, especially the ones coming out of China. The GLM, Qwen and DeepSeek models out of C…

Not only have I been following a lot of the open models, you may find it surprising I have extensively tested some of them and coerced them to generate deterministic responses across different machines as a method to prove responses are not tampered with, as well as developing ways to run them in remotely attestable secure enclaves to ensure people that use them for sensitive applications can have provable privacy wi…

I respect this, you've clearly done your due diligence here.
Post reply on HN