Claude can now search the web • Hacker News Board

Re: Claude can now search the web

#21

Post by Demiurge » Thu, Mar 20, 2025, 5:10 PM UTC

micromacrofoot wrote: post #14

NBJack wrote: post #10

I wonder if it will actually respect the robots.txt this time.

almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off

In what circles is it a joke? Google bots seem to respect it on my sites according to logs.

Re: Claude can now search the web

#22

Post by explain » Thu, Mar 20, 2025, 5:12 PM UTC

NBJack wrote: post #10

I wonder if it will actually respect the robots.txt this time.

robots.txt is meant for automated crawlers, not human-driven actions.

Re: Claude can now search the web

#23

Post by callamdelaney » Thu, Mar 20, 2025, 5:12 PM UTC

Brusco_RF wrote: post #13

newfocogi wrote: post #8

Earlier quoted context omitted.

ChatGPT can search the web (I just checked in my account). It appears to be available for all users (not just paid): https://help.openai.com/en/articles/9237897-chatgpt-search

I definitely tried to web search with ChatGPT a few weeks ago and it couldn't. I don't think I'm making this up. Unless I suffered a TBI.

It told me it can't search the web and then proceeded to search the web

Re: Claude can now search the web

#24

Post by mvieira38 » Thu, Mar 20, 2025, 5:12 PM UTC

What's up with the geoblocking of Claude features? Not the first time it happens

Re: Claude can now search the web

#25

Post by geekrax » Thu, Mar 20, 2025, 5:13 PM UTC

micromacrofoot wrote: post #14

NBJack wrote: post #10

I wonder if it will actually respect the robots.txt this time.

almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off

I have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.

Re: Claude can now search the web

#26

Post by itpcc » Thu, Mar 20, 2025, 5:13 PM UTC

Now I understand why Gitlab was (is?) attacked[0] by those hideous bots.

[0] https://news.ycombinator.com/item?id=43422413

Re: Claude can now search the web

#27

Post by pcj-github » Thu, Mar 20, 2025, 5:14 PM UTC

Does not really say /how/ it's performing a web search... Is it tapping into it's "own" corpus of material or calling out to some other web search engine?

Re: Claude can now search the web

#28

Post by otikik » Thu, Mar 20, 2025, 5:17 PM UTC

Demiurge wrote: post #21

micromacrofoot wrote: post #14

Earlier quoted context omitted.

almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off

In what circles is it a joke? Google bots seem to respect it on my sites according to logs.

Apparently, the regular search crawler does it, but the ai thingie doesn't.

Re: Claude can now search the web

#29

Post by punkpeye » Thu, Mar 20, 2025, 5:17 PM UTC

There is already a 100 ways of doing it using MCP

https://glama.ai/mcp/servers?searchTerm=search

What's the benefit of bringing native integration?

Re: Claude can now search the web

#30

Post by light_triad » Thu, Mar 20, 2025, 5:17 PM UTC

Good news. I integrated Claude with a scrapper to get info from pages and it was not giving hallucinations 99% of the time. Hope this works out of the box now.