Live data from Hacker News

How much of AI's recent success is due to the Forer Effect?

shkspr.mobi

31–40 of 83 posts

Re: How much of AI's recent success is due to the Forer Effect?

#31

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

The compilers _usually_ generate correct machine instructions (code) better than us. This, alone, is not really a meaningful measure and doesn’t refute the claim.

Re: How much of AI's recent success is due to the Forer Effect?

#32
I remember I was working on a game where you fly a ship and shoot alien ships. I didnt have a lot of experience at the time so I just made the ships move in one direction and eventually take a random turn, but if you got within a distance they moved away from you, but if you got too close they would kamikaze toward you. but I needed to figure out when to shoot, so I just decided that they would randomly shoot when you shoot.

However, what happened is that since the distance of your bullets is small, you would fly up towards them and they would 'trick' you by at first flying away then suddenly b-lining at you. and when you fired sometimes they would fire at the same time and get you.

The comments I got from people who played the game 'wow the AI is really good'. Not sure if this is a direct example of the Forer effect

Re: How much of AI's recent success is due to the Forer Effect?

#33

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

Do you mind sharing what you're using / how you're doing it?

I've been using github copilot and I'm waving back and forth between impressed and unimpressed; it regularly generates syntactically invalid code for me. Eg calling lib functions that don't exist.

Thanks!

Re: How much of AI's recent success is due to the Forer Effect?

#34

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

Counter point that literally just bit me this morning. ChatGPT completely lied to me about the scaling properties of Kinesis. I was confused about the 1000 write / second but only 5 read / second throughput of a kinesis shard and was asking it a number of questions, including specifically if a batch read would count as only a single read operation. It explicitly told me that each record in the batch counted as an ind…

Yes, please. Do not use LLMs as a substitute for Google search. If you are looking for factual information, just use google, bing, or duckduckgo. You should only use ChatGPT for things that you are able to review it's work.

Technology is supposed to make us smarter. Blindly believing an AI that we know can hallucinate makes us dumb with confidence.

Re: How much of AI's recent success is due to the Forer Effect?

#35
post #27
post #19

Earlier quoted context omitted.

That’s not really an issue when we’re talking about AI. I said in a separate comment on the topic: intelligence is such a complicated thing that we seem to only be able to define it by pointing to things and saying “that’s not it.” If we didn’t move the goalposts, we’d have declared Stockfish to be full AI, despite it only being a chess-playing program, long ago.

Except in a few years you may realize it was "it" all along.

Perhaps but I’m pretty confident a ChatGPT is not “it” this time.

Re: How much of AI's recent success is due to the Forer Effect?

#36
post #13

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

Let me make this very clear: Generating syntactically correct code in various languages that also looks plausible is no small feat. It is, in fact, extremely impressive and will certainly have an impact on SE. But. Every single test I ran lead to functionally wrong designs from smallish memory errors in C (that hilariously ChatGPT was able to correct ND explain when pointed to) over misplaced/hallucinated methods in…

I can guarantee you that the equivalent code which I generate (taking >100 times as much time) has a comparable error/typo rate. I can say this with confidence because ChatGPT, while not perfect, "knows" lots of things I don't know and absolutely makes me 10x as efficient. Now, I am a pretty crummy software engineer so ymmv, but it's worth noting how far the goalposts have moved for assessing AI: from "simulates understanding of language in a general way" to "comparable in any way to a professional specialist in a random task (and much faster)". A few months ago everyone was losing their minds over how good ChatGPT was, and while some of that was overblown, nothing has fundamentally changed except we have become used to it and are looking for the next shiney tech thing. At least we're not talking about crypto anymore.

Re: How much of AI's recent success is due to the Forer Effect?

#37
post #13

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

Let me make this very clear: Generating syntactically correct code in various languages that also looks plausible is no small feat. It is, in fact, extremely impressive and will certainly have an impact on SE. But. Every single test I ran lead to functionally wrong designs from smallish memory errors in C (that hilariously ChatGPT was able to correct ND explain when pointed to) over misplaced/hallucinated methods in…

I have the same experience. I think the transformer-based models are impressive, but whenever I try to actually use them for something, e.g. having ChatGPT write something or using GitHub Copilot, the results I get are terrible.

I was legitimately afraid for my job when GPT-4 was released. After using it, however, I sleep easy now.

Re: How much of AI's recent success is due to the Forer Effect?

#38
post #16

I agree with the other comments that this is a fairly useless article. Perhaps I’m looking too much into this specific example, but I fail to see how the entirety of AI’s success (per this title) is misrepresented due to its inability to provide a horoscope (good example by another comment) that’s specific to some person in question. We see a lot of articles that swing too far towards “AI will change everything!” jus…

> I fail to see how the entirety of AI’s success (per this title) is misrepresented

The title does not claim that the "entirety" of AI's success is misrepresented. It is questioning "how much" is, and I think that this is a fair question. The author does admit that the first paragraph in the example does have valid information which shows the AI with knowledge. It's the second paragraph that is just fluff.

I think that it is valid to ask how much this "fluff" impresses us and leads us to see the AI as being knowledgeable. Perhaps the author does draw too strong of a conclusion, but he still makes a good point.

Re: How much of AI's recent success is due to the Forer Effect?

#39
post #37
post #13

Earlier quoted context omitted.

Let me make this very clear: Generating syntactically correct code in various languages that also looks plausible is no small feat. It is, in fact, extremely impressive and will certainly have an impact on SE. But. Every single test I ran lead to functionally wrong designs from smallish memory errors in C (that hilariously ChatGPT was able to correct ND explain when pointed to) over misplaced/hallucinated methods in…

I have the same experience. I think the transformer-based models are impressive, but whenever I try to actually use them for something, e.g. having ChatGPT write something or using GitHub Copilot, the results I get are terrible. I was legitimately afraid for my job when GPT-4 was released. After using it, however, I sleep easy now.

I had the same experience, I was worried and now that I've used the tools I sleep easier. But I still worry that these tools will improve dramatically and quickly. Somebody tell me I don't have to worry, please.

Re: How much of AI's recent success is due to the Forer Effect?

#40

The Forer Effect can't make chatGPT generate syntactically correct and functional code snippets that I use every day at work.

Came here to say this. The code it generates for me is usually not perfect, but for me it provides conceptual insights that have gotten me unstuck from gnarly situations.

The conventional narrative is that chatGPT can code as well as a junior engineer, but I feel like it’s more like a senior engineer scribbling on a whiteboard—no, it’s not perfect code, but it’s the right idea.

Post reply on HN