Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude can now search the web

anthropic.com

Post reply on HN
Print view View original

71–80 of 758 posts

Re: Claude can now search the web

#71

Post by 1shooner »

_2d30 wrote: post #39
NBJack wrote: post #10

I wonder if it will actually respect the robots.txt this time.

I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.

>You can now use Claude to search the internet to provide more up-to-date and relevant responses.

It's a search engine. You 'ask it to read the web' just like you asked Google to, except Google used to actually give the website traffic.

I appreciate the concept of an AI User-agent, but without a business model that pays for the content creation, this is just going to lead to the death of anonymously accessible content.

Re: Claude can now search the web

#73

Post by Joe_Cool »

geekrax wrote: post #25
micromacrofoot wrote: post #14

Earlier quoted context omitted.

almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off

I have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.

For my personal stuff I also included a Nepenthes tarpit. Works great and slows the bots down while feeding them garbage. Not my fault when they consume stuff robots.txt says they shouldn't.

I'm just not sure if legal would love me doing that on our corporate servers...

Re: Claude can now search the web

#75

Post by darepublic »

1shooner wrote: post #71
_2d30 wrote: post #39

Earlier quoted context omitted.

I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.

>You can now use Claude to search the internet to provide more up-to-date and relevant responses. It's a search engine. You 'ask it to read the web' just like you asked Google to, except Google used to actually give the website traffic. I appreciate the concept of an AI User-agent, but without a business model that pays for the content creation, this is just going to lead to the death of anonymously accessible conten…

Well I expect eventually the agent will be able to act on your behalf with your credentials.

Re: Claude can now search the web

#78

Post by TheDudeMan »

mvdtnz wrote: post #68
_2d30 wrote: post #39

Earlier quoted context omitted.

I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.

No thank you, when I define a robots.txt file I expect all automated systems to respect it.

But this isn't automated. This is user-driven.

Re: Claude can now search the web

#80

Post by masterj »

punkpeye wrote: post #42
TrueDuality wrote: post #34

Earlier quoted context omitted.

The native app that allows for MCP is only available officially on Mac's and the web interface is generally more convenient for non-technical users. Searching and interacting with the web has become a table-stakes feature and was a glaring gap in Claude.

Let me rephrase it. MCP has the capability to add this functionality. It would be nice to see MCP getting adoption in their web UI, as well easier UX, rather than more ad hoc features being added natively.

This is likely implemented behind the scenes as an MCP server exposed to their model in the web UI. It is likely that they will enable MCP servers over HTTP+SSEs (vs the stdin/stdout used with Claude desktop) on the web version in the near future.
Post reply on HN