Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

simonwillison.net

Post reply on HN
Print view View original

241–250 of 268 posts

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#241

Post by herewegohawks »

Helmut10001 wrote: post #213
larsiusprime wrote: post #93

I find ChatGPT to be great at research too-but there are pathological failure modes where it is biased to shallow answers that are subtly wrong, even when definitive primary sources are readily available online: https://www.fortressofdoors.com/researchers-beware-of-chatgp...

More recently, I find ChatGPT to become increasingly unreliable. It makes up almost every second answer, forgets context, or is just downright wrong. Maybe I am used these days more and more to dump huge texts for context into the prompt, as aistudio allows me. Maybe ChatGPT isn't as good as with such information. Gemini/Aistudio will stay on track even with 300k tokens consumed, it just needs a little nudge here and…

FWIW, I found things improved greatly once I turned off the memory feature of ChatGPT. My guess is that a lot of tokens were going towards trying to follow instructions from past conversations.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#242

Post by magicalist »

simonw wrote: post #238
larsiusprime wrote: post #236

Earlier quoted context omitted.

My contention is if it’s going to just give me a Wikipedia summary, I can do that myself. I just have greater expectations of “PhD” level intelligence. If we’re going to claim to it is PhD level it should be able to do “deep” research AND think critically about source credibility, just as a PhD would. If it can’t do that they shouldn’t brand it that way. Also it’s not like I’m taking Matzat’s word for anything. I can…

I suggest ignoring the "PhD level intelligence" marketing hype.

A couple of times when I've gotten an answer sourced basically only from wikipedia and stackoverflow, I've thrown in a comment about its "PhD level intelligence" when I tell it to dig deeper, and it's taken it pretty well ("fair jab :)"), which is amusing. I guess that marketing term has been around long enough to be in gpt5's training data.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#243

Post by lambda »

simonw wrote: post #166
lambda wrote: post #157

I guess the part where I'm still skeptical are: Google is also still pretty good at search (especially if I avoid the AI summary with udm=14). I'll take one of your examples: Britannica to seed Wikipedia. I searched for "wikipedia encyclopedia brtannica". In less than 1 second, I got search results back. I spend maybe 30 seconds scanning the page; past the Wikipedia article on Encyclopedia Britannica, past the Encycl…

I suggest trying that experiment again but picking the hardest of my examples to answer with Google, not the easiest.

Not sure which is the hardest, but sure, let's try them all.

* Bouncy people mover. Some Google searching turns up the SFO article that you liked. Trying to pin down the exact dates is harder. ChatGPT maybe did narrow down the time frame quicker than I could through a series of Google searches,

* The picture of the building. Go to Google lens, paste in the image, less than a second later I get results. Of course, the exact picture in this article comes up on top, but among the other results I get a mix of two different buildings, one of which is identified as the Blade, one Independence Temple. So a few seconds here between searching and doing my own quick visual scan of the results.

* Starbucks UK Cake Pops: This one is harder to find the full details with a quick Google search. I am able to find that the were fairly recently introduced in the UK after my second search. It looks like ChatGPT gave you a bunch of extra response, some of which you didn't like, because you then spent a while trying to reverse engineer its system prompt rather than any actual follow up on the question itself.

* Official name of the University of Cambrdige: search gave me Wikipedia, Wikipedia contains the official name and a link to a reference on the University's page. Pretty quick to solve with Google Search/Wikipedia.

* Exeter quay. I searched for "waterfront exeter cliff building" and found this result towards the top of the results: https://www.exeterquay.org/milestones/ which explains "Warehouses were added in 1834 [Cornish's] and 1835 [Hooper's], with provision for storing tobacco and wine and cellars for cider and silk were cut into the cliffs downstream." You seemed to be a lot more entertained by ChatGPT's persistence in finding more info, but for satisfying curiosity about the basic question, I got an answer pretty quickly via Google.

* Aldi vs Lidl: this is a much more subjective question, so whether the results you get via a quick Google search meet your needs, vs. whether the summary of subjective results you get via ChatGPT, is more of a question you can answer. I do find some Reddit threads and similar with a quick Google search.

* Book scanning. You asked specifically about destructive book scanning. You can do a quick search of each of the labs and "book scanning" and find the same lack of results that ChatGPT gives you. Maybe takes a similar amount of time to how long it spent thinking. You pretty much only find references to Anthropic doing destructive book scanning, and Google doing mostly non-destructive scanning

Anyhow, the results are mixed. For a bunch of these, I found an answer quicker via a Google search (or Google Lens search), and doing some quick scanning/filtering myself. A few of them, I feel like it was a wash. A couple of them actually do take more iteration/research, the bouncy travelator being the most extreme example, I think; narrowing down the timeline on my own would take a lot of detailed looking through sources.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#244

Post by typpilol »

simonw wrote: post #165
typpilol wrote: post #142

Earlier quoted context omitted.

Yes this feels very AI like too. A ton of prose for very little substance lol. I skipped half the article to get to the point, went back and re-read and didn't miss much.

I don't use AI to generate writing on my blog.

Oh I know. Just this particular article seemed very sparse.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#245

Post by Atotalnoob »

Moosdijk wrote: post #185
NothingAboutAny wrote: post #170

Earlier quoted context omitted.

I tried to use perplexity to find ideal settings for my monitor, it responded with concise list of distinct settings and why. When I investigated the source it was just people guessing and arguing with each other in the Samsung forums, no official or even backed up information. I'd love if it had a confidence rating based on the sources it found or something, but I imagine that would be really difficult to get right.

I asked gemini to do a deep research on the role of healthcare insurance companies in the decline of general practicioners in the Netherlands. It based its premise mostly on blogs and whitepapers on company websites, who's job it is to sell automation-software. AI really needs better source-validation. Not just to combat the hallucination of sources (which gemini seems to do 80% of the time), but also to combat low q…

Kagi has some tooling for this. You can set web access “lenses” that limit the results to “academic”, “forums”, etc.

Kagi also tells you the percentages “used” for each source and cites them in line.

It’s not perfect, but it’s a lot better to narrow down what you want to get out of your prompt.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#246

Post by Faaak »

lambda wrote: post #157

I guess the part where I'm still skeptical are: Google is also still pretty good at search (especially if I avoid the AI summary with udm=14). I'll take one of your examples: Britannica to seed Wikipedia. I searched for "wikipedia encyclopedia brtannica". In less than 1 second, I got search results back. I spend maybe 30 seconds scanning the page; past the Wikipedia article on Encyclopedia Britannica, past the Encycl…

A bit unrelated, but on Firefox there's the Straight to the Web extension that automatically appends the udm=14 param, so AI gets disabled :-)

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#247

Post by bitexploder »

bluecalm wrote: post #181
losvedir wrote: post #117

Earlier quoted context omitted.

I dunno, I use Deep Research from Claude, ChatGPT, and Gemini, and Gemini is the only one that ignores my requests and always produces the most inane high school student wannabe management consultant "report" with introduction and restatement of the problem and background and all that. Its "voice" (the prose, I mean, not text to speech) is so irritating I've stopped using it. The other ones will do the thing I want:…

Gemini is high on hallucination. When I ask it about my own software it not only changes my own name to a similar one common in my language but also makes up stuff about our team saying some stranger works with us (he works in the same niche but that's about it). It's annoying when it's so confident making up nonsense. Imo Chat GPT is just a league above when it comes to reliability.

I just end up using both for research type things. They both end up doing better on certain topics or types of work. For $20/mo why not both :)

I like ChatGPT as a product more, but Gemini does well on many things that ChatGPT struggles with a little more. Just my anecdotes.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#248

Post by prmoustache »

sigmoid10 wrote: post #173
wilg wrote: post #172

Earlier quoted context omitted.

First, you not having to spend the 60 seconds and it means you can parallelize it with something else to get the answer effectively instantly. Second, you're essentially establishing that if an LLM can get it done in less than 60 seconds its better than your manual approach, which is a huge win, as this will get faster!

For real. This is what it must have been like living in the early 20th century and hearing people say they prefer a horse to get groceries because it is so much more effort to crank-start a car. I look forward to the age when we gleefully reminisce about the time we had to deal with SEO spam manually.

The thing is, the idea of cars cars being a plus for humanity is still debatable to this day. We won a lot in some areas but lost an awful lot in others.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#249

Post by lambda »

wilg wrote: post #172
lambda wrote: post #157

I guess the part where I'm still skeptical are: Google is also still pretty good at search (especially if I avoid the AI summary with udm=14). I'll take one of your examples: Britannica to seed Wikipedia. I searched for "wikipedia encyclopedia brtannica". In less than 1 second, I got search results back. I spend maybe 30 seconds scanning the page; past the Wikipedia article on Encyclopedia Britannica, past the Encycl…

First, you not having to spend the 60 seconds and it means you can parallelize it with something else to get the answer effectively instantly. Second, you're essentially establishing that if an LLM can get it done in less than 60 seconds its better than your manual approach, which is a huge win, as this will get faster!

There's no useful parallelization that could happen during this particular search. This took a couple of iterations of research via ChatGPT, and then reading the results and looking at the referenced sources; the total interaction time with ChatGPT is a similar 60 seconds or so, the main difference is the 3 minutes of waiting for it to generate answers vs. the maybe a couple of seconds for the searches.

Re: GPT-5 Thinking in ChatGPT (a.k.a. Research Goblin) is good at search

#250

Post by recursive »

rossant wrote: post #215
gitmagic wrote: post #184

Earlier quoted context omitted.

I’ve been thinking about this as well. Instead of making websites, maybe people will make something else, like some future version of MCP tools/servers? E.g. a restaurant could have an “MCP tool” for checking opening hours, reserving a table, etc.

Same. Websites won't disappear but may become niche or something of the past. Why create a new UI for your new service when you can plug into a "universal" personal agent AI.

So I can get reliable answers.

Why create a fancy infrastructure for for this new universal thing, when the old thing already does it more reliably and with less steps?

Post reply on HN