AI assisted search-based research works now • Hacker News Board

Re: AI assisted search-based research works now

#101

Post by ljsprague » Tue, Apr 22, 2025, 9:13 AM UTC

riku_iki wrote: post #84

CSMastermind wrote: post #79

Earlier quoted context omitted.

544 rosters but half as many games (because the teams play each other). Technically I can probably do it in about 10 minutes because I've worked with these kind of stats before and know about packages that will get you this basically instantly ( https://pypi.org/project/nfl-data-py/ ). It's exactly 4 lines of code to find the correct answer, which is 2,227. Assuming I didn't know about that package though I'd open a…

https://chatgpt.com/share/6807278c-a0d0-8006-80f3-f62ae9f8ff...

Did you run the code?

Re: AI assisted search-based research works now

#102

Post by ljsprague » Tue, Apr 22, 2025, 9:18 AM UTC

xp84 wrote: post #54

From article: > “Google is still showing slop for Encanto 2!” (Link is provided) I believe quite strongly that Google is making a serious misstep in this area, the “supposed answer text pinned at the top above the actual search results.” For years they showed something in this area which was directly quoted from what I assume was a shortlist of non-BS sites so users were conditioned for years that if they just wanted…

I have stopped using Google when I have a random fact I need answered. Faster to ask ChatGPT. I trust it enough now.

Re: AI assisted search-based research works now

#103

Post by spongebobstoes » Tue, Apr 22, 2025, 9:44 AM UTC

otistravel wrote: post #94

The most impressive demos of these tools always involve technical tasks where the user already knows enough to verify accuracy. But for the average person asking about health issues, legal questions, or historical facts? It's basically fancy snake oil - confident-sounding BS that people can't verify. The real breakthrough would be systems that are actually trustworthy without human verification, not slightly better B…

this remains true for pretty much all advice or information we receive. doctors, lawyers, accountants, teachers. there have been countless times that all of these professionals have given me bad advice or information

sure, at least I have someone to blame in that case. but in my experience, the AI is at least as reliable as a person who I don't personally know

Re: AI assisted search-based research works now

#104

Post by red_trumpet » Tue, Apr 22, 2025, 9:44 AM UTC

minraws wrote: post #89

simonw wrote: post #58

Earlier quoted context omitted.

That is an excellent prompt to tuck away in your back pocket and try again future iterations of this technology. It's going to be an interesting milestone when or if any of these systems get good enough at comprehensive research to provide a correct answer.

If you keep the prompt the same at some point the data will appear in training set and we might have answer. So even though today it might be a good check it might not remain as such a good benchmark. I think we need a way to keep updating prompts without increasing complexity in someway to properly verify model improvements. ARC Deep Research anyone?

Well, to test research capabilities, one could just adopt the year (2024->2025) in the prompt.

Re: AI assisted search-based research works now

#105

Post by csallen » Tue, Apr 22, 2025, 10:15 AM UTC

It's actually quite doable to build your own deep research agent. You just need a single prompt, a solid code loop to run it agentically, and some tools for it to call. I've been building a domain-specific deep research agent over the past few days for internal use, and I'm pretty impressed with how much better it is than any of the official deep search agents for my use case.

Re: AI assisted search-based research works now

#106

Post by patapong » Tue, Apr 22, 2025, 10:39 AM UTC

wontonaroo wrote: post #91

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

I used Google AI Studio instead of Google Gemini App because it provides references to the search results. Google AI Studio gave me an exact answer of 2227 as a possible answer and linked to these comments because there is a comment further down which claims that is the exact answer. The comment was 2 hours old when I did the prompt. It also provided a code example of how to find it using the python nfl data library…

So the time to test data leakage from posting a question and answer to the internet, to LLMs having access to the answer is less than 2h... Does not bode well for the benchmarks of the future!

Re: AI assisted search-based research works now

#107

Post by neural_thing » Tue, Apr 22, 2025, 11:16 AM UTC

spongebobstoes wrote: post #103

otistravel wrote: post #94

The most impressive demos of these tools always involve technical tasks where the user already knows enough to verify accuracy. But for the average person asking about health issues, legal questions, or historical facts? It's basically fancy snake oil - confident-sounding BS that people can't verify. The real breakthrough would be systems that are actually trustworthy without human verification, not slightly better B…

this remains true for pretty much all advice or information we receive. doctors, lawyers, accountants, teachers. there have been countless times that all of these professionals have given me bad advice or information sure, at least I have someone to blame in that case. but in my experience, the AI is at least as reliable as a person who I don't personally know

I tested o3 on a medical issue I've had that 50+ doctors couldn't diagnose over the span of 6-7 years, ended up figuring it out through sheer luck. With my first prompt, it gave a list of probabilities, with the correct answer being listed as the third most likely. It also suggested correct tests to run for every option. I trust it way more than I trust human doctors who were confidently wrong about me for years.

Re: AI assisted search-based research works now

#108

Post by mehulashah » Tue, Apr 22, 2025, 11:18 AM UTC

I find that people often conflate search with analytics when discussing Deep Research. Deep Research is iterated search and tool use. And, no doubt, it’s remarkably good. Deep Analytics is like Deep Research in that it uses generative AI models to generate a plan, but LLMs operations and structured (tool use) are interleaved in database style query pipelines. This allows for the more precise counting and exhaustive search type use cases.

Re: AI assisted search-based research works now

#109

Post by JustFinishedBSG » Tue, Apr 22, 2025, 11:26 AM UTC

AlotOfReading wrote: post #46

sshine wrote: post #32

Earlier quoted context omitted.

Kagi is $10/mo. for search and +$15/mo. for premium LLMs with agentic access to search.

What they're talking about is access to professional archives like EBSCOnet or Bloomberg, which usually don't sell to individuals in the first place and start at tens of thousands of dollars per seat for institutional access.

Ofc Mistral model is a lot worse but for $14.99 you get access to AFP news. So OpenAI for $60 for the same things would be a huge joke

Re: AI assisted search-based research works now

#110

Post by pyfon » Tue, Apr 22, 2025, 11:27 AM UTC

jonas_b wrote: post #99

A common google searching thing I counter have is something like this: I need to get from A to B via C via public transport in a big metropolis. Now C could be one of say 5 different locations of a bank branch, electronics retailer, blood test lab or whatever, so there's multiple ways of going about this. I would like a chatbot solution that compares all the different options and lays them out ranked by time from A t…

I am on holiday now and want something similar. Get me from A to B but with a memorable heuristic that I can use if I leave at any time. E.g. "if you catch a 134 or 175 bus to Kings station then get the metro 3 stops to Cental station.". Even better if you add some landmarks.

This may exclude some clever routes that shave off 3 minutes if you do the correct parkour... but it means I can now put my phone down and enjoy the journey without tracking it like a hawk.