AI assisted search-based research works now • Hacker News Board

Re: AI assisted search-based research works now

#51

Post by rd » Mon, Apr 21, 2025, 9:37 PM UTC

simonw wrote: post #49

hadlock wrote: post #47

Earlier quoted context omitted.

o3/o4 seem to know how to search things like pypi, crates.io, pkg.go.dev etc and apply those changes on the first try. My application (running on an older version of code) had a breaking change to how the event controller functioned in the newer version, o3 looked at the documentation and rewrote it to use the new event controller. It used to be that you were trapped with the LLM being 3-8 months behind on package ve…

Huh, now I'm thinking that maybe a target for release notes should be to provide enough details that a good LLM can be used to apply fixes for any breaking changes.

MCP maybe? A release notes MCP (maybe into ReadTheDocs or pypi) that understands upgrade instructions for every package.

Re: AI assisted search-based research works now

#52

Post by sanderjd » Mon, Apr 21, 2025, 9:56 PM UTC

rd wrote: post #51

simonw wrote: post #49

Earlier quoted context omitted.

Huh, now I'm thinking that maybe a target for release notes should be to provide enough details that a good LLM can be used to apply fixes for any breaking changes.

MCP maybe? A release notes MCP (maybe into ReadTheDocs or pypi) that understands upgrade instructions for every package.

This is the thing I don't really love about MCP: Why should it require a separate protocol, rather than just good readable documentation?

Re: AI assisted search-based research works now

#53

Post by 63 » Mon, Apr 21, 2025, 10:22 PM UTC

One downside I found is that the llm cannot change its initial prompt until it's done thinking. I used deep research to compare counseling centers for me but of course when it encounters some factor I hadn't thought of (e.g. the counselors here fit the criteria perfectly but none accept my insurance), it doesn't know that it ought to skip that site entirely. Really this is a critique of the deep-research approach rather than search in general, but I imagine it can still play out on smaller scales. Often, searching for information is a dynamic process involving the discovery of unknown unknowns and adjustment based on that, but ai isn't great at abstract goals or stopping to ask clarifying questions before resuming. Ultimately, the report I got wasn't useless, but it mostly just regurgitated the top 3 google results. I got much better recommendations by reaching out to a friend who works in the field.

Re: AI assisted search-based research works now

#54

Post by xp84 » Mon, Apr 21, 2025, 10:24 PM UTC

From article:

> “Google is still showing slop for Encanto 2!” (Link is provided)

I believe quite strongly that Google is making a serious misstep in this area, the “supposed answer text pinned at the top above the actual search results.”

For years they showed something in this area which was directly quoted from what I assume was a shortlist of non-BS sites so users were conditioned for years that if they just wanted a simple answer like when a certain movie came out or if a certain show had been canceled or something, you may as well trust it.

Now it seems like they have given over that previous real estate to a far less reliable feature, which simply feeds any old garbage it finds anywhere into a credulous LLM and takes whatever pops out. 90% of people that I witness using Google today simply read that text and never click any results.

As a result, Google is now pretty much always even less accurate at the job of answering questions than if you posed that same question to ChatGPT, because GPT seems to be drawing from its overall weights which tend toward basic reality, whereas Google’s “Answer” seems to be summarizing a random 1-5 articles from the Spam Web, with zero discrimination between fact, satire, fiction, and propaganda. How can they keep doing this and not expect it to go badly?

Re: AI assisted search-based research works now

#55

Post by navinsylvester » Mon, Apr 21, 2025, 10:25 PM UTC

sanderjd wrote: post #52

rd wrote: post #51

Earlier quoted context omitted.

MCP maybe? A release notes MCP (maybe into ReadTheDocs or pypi) that understands upgrade instructions for every package.

This is the thing I don't really love about MCP: Why should it require a separate protocol, rather than just good readable documentation?

Ironically, https://context7.com/

Re: AI assisted search-based research works now

#56

Post by fudged71 » Mon, Apr 21, 2025, 10:29 PM UTC

swyx wrote: post #36

> Deep Research, from three different vendors dont forget Xai grok!

you.com is surprisingly good for this as well (I like the corporate report PDF export)

Re: AI assisted search-based research works now

#57

Post by CSMastermind » Mon, Apr 21, 2025, 10:35 PM UTC

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate."

I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for me to find this number. It might take an hour of my time.

But despite this being a relatively easy research task all of the deep research tools I tried (OpenAI, Google, and Perplexity) completely failed and just gave me a general estimate.

Based on this article I tried that search just using o3 without deep research and it still failed miserably.

Re: AI assisted search-based research works now

#58

Post by simonw » Mon, Apr 21, 2025, 10:50 PM UTC

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

That is an excellent prompt to tuck away in your back pocket and try again future iterations of this technology. It's going to be an interesting milestone when or if any of these systems get good enough at comprehensive research to provide a correct answer.

Re: AI assisted search-based research works now

#59

Post by danielmarkbruce » Mon, Apr 21, 2025, 10:52 PM UTC

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

This is just a bad match to the capabilities. What you are actually looking for is analysis, similar in nature to what a data scientist may do.

The deep research capabilities are much better suited to more qualitative research / aggregation.

Re: AI assisted search-based research works now

#60

Post by [deleted] » Mon, Apr 21, 2025, 10:55 PM UTC

danielmarkbruce wrote: post #59

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

This is just a bad match to the capabilities. What you are actually looking for is analysis, similar in nature to what a data scientist may do. The deep research capabilities are much better suited to more qualitative research / aggregation.

[deleted]