Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assisted search-based research works now

simonwillison.net

Post reply on HN
Print view View original

81–90 of 156 posts

Re: AI assisted search-based research works now

#81

Post by jsemrau »

simonw wrote: post #7
jsemrau wrote: post #3

My main observation here is 1. Technically it might be possible to search the Internet, but it might not surface correct and/or useful information. 2. High-value information that would make a research report valuable is rarely public nor free. This holds especially true in capital-intensive or regulated industries.

I fully expect one of the AI-related business models going forward to be charging subscriptions for LLM search tool access to those kinds of archives. ChatGPT plus an extra $30/month for search access to a specific archive would make sense to me.

Then I'd rather see domain-specific agent-first data. I.e., not a simple API call but token->BM25->token

Re: AI assisted search-based research works now

#82

Post by jsemrau »

TrackerFF wrote: post #48
jsemrau wrote: post #3

My main observation here is 1. Technically it might be possible to search the Internet, but it might not surface correct and/or useful information. 2. High-value information that would make a research report valuable is rarely public nor free. This holds especially true in capital-intensive or regulated industries.

I'm not a researcher, but don't most researchers these days also upload their work to arXiv? Sure, it's not a journal - but in some fields (Machine Learning, Math) it seems like everyone also uploads their stuff there. So if the models can crawls sites like arXiv, at least there's some decent stuff to be found.

Proper research, especially the one contributed to conferences, is hard to get by and is usually managed by the conference organizers. Arxiv has some, but it's limited.

It would be great if for a DeepSearch tool for ML, I could just use Arxiv as a source and have the Agent search this. But so far I have not found a working Arxiv tool that does this well.

Re: AI assisted search-based research works now

#83

Post by [deleted] »

jeffbee wrote: post #72

The Deep Research stuff is crazy good. It solves the issue that I can often no longer find articles that I know are out there. Example: yesterday I was holding forth on the socials about how 25 years ago my local government did such and such thing to screw up an apartment development at the site of an old movie theater, but I couldn't think of the names of any of the principals. After Googling for a bit I used a Deep…

[deleted]

Re: AI assisted search-based research works now

#84

Post by riku_iki »

CSMastermind wrote: post #79
neom wrote: post #64

Earlier quoted context omitted.

Is it accurate that there are 544 rosters? If so, even at 2 minutes a roster isn't that days of work, even if you coded something? How would you go about completing this task in 1 hour as a human? (also chatgpt 4.1 gave me 2,503 and it said it used the NFL 2024 fact book)

544 rosters but half as many games (because the teams play each other). Technically I can probably do it in about 10 minutes because I've worked with these kind of stats before and know about packages that will get you this basically instantly ( https://pypi.org/project/nfl-data-py/ ). It's exactly 4 lines of code to find the correct answer, which is 2,227. Assuming I didn't know about that package though I'd open a…

https://chatgpt.com/share/6807278c-a0d0-8006-80f3-f62ae9f8ff...

Re: AI assisted search-based research works now

#85

Post by johnnyanmac »

danielmarkbruce wrote: post #59
CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

This is just a bad match to the capabilities. What you are actually looking for is analysis, similar in nature to what a data scientist may do. The deep research capabilities are much better suited to more qualitative research / aggregation.

If AI Can't look up and read a chart, why would I trust it with any real aggregation?

Re: AI assisted search-based research works now

#86

Post by kenjackson »

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

o3 deep research gave me an answer after I requested an exact answer again (it gave me an estimate first): 2147.

Re: AI assisted search-based research works now

#87

Post by intended »

ilrwbwrkhv wrote: post #63
intended wrote: post #12

I find that these conversations on HN end up covering similar positions constantly. I believe that most positions are resolved if 1) you accept that these are fundamentally narrative tools. They build stories, In whatever style you wish. Stories of code, stories of project reports. Stories or conversations. 2) this is balanced by the idea that the core of everything in our shared information economy is Verification.…

Yup succinct summarization of the current state. This works across domains from research to software engineering.

New LinkedIn Post.

Re: AI assisted search-based research works now

#88

Post by raybb »

CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

Similarly, I asked it a rather simple question of giving me a list of AC repair places near me with my numbers. Weirdly, Gemini repeated a bunch of them like 3 or 4 times, gave some completely wrong phone numbers, and found many places hours away but labeled them as in the neighboring city.

Re: AI assisted search-based research works now

#89

Post by minraws »

simonw wrote: post #58
CSMastermind wrote: post #57

The various deep research products don't work well for me. For example I asked these tools yesterday, "How many unique NFL players were on the roster for at least one regular season game during the 2024 season? I'd like the specific number not a general estimate." I as a human know how to find this information. The game day rosters for many NFL teams are available on many sites. It would be tedious but possible for m…

That is an excellent prompt to tuck away in your back pocket and try again future iterations of this technology. It's going to be an interesting milestone when or if any of these systems get good enough at comprehensive research to provide a correct answer.

If you keep the prompt the same at some point the data will appear in training set and we might have answer.

So even though today it might be a good check it might not remain as such a good benchmark.

I think we need a way to keep updating prompts without increasing complexity in someway to properly verify model improvements. ARC Deep Research anyone?

Re: AI assisted search-based research works now

#90

Post by netghost »

johnnyanmac wrote: post #85
danielmarkbruce wrote: post #59

Earlier quoted context omitted.

This is just a bad match to the capabilities. What you are actually looking for is analysis, similar in nature to what a data scientist may do. The deep research capabilities are much better suited to more qualitative research / aggregation.

If AI Can't look up and read a chart, why would I trust it with any real aggregation?

Because AI is weird and does some things really well, and some things poorly. The terrible/exciting/weird part is figuring out which is which.
Post reply on HN