I haven't used Claude yet, but heard many good things. So I'm surprised to see that they're so far behind on this feature.
Claude can now search the web
41–50 of 758 posts
Re: Claude can now search the web
#42There is already a 100 ways of doing it using MCP https://glama.ai/mcp/servers?searchTerm=search What's the benefit of bringing native integration?
The native app that allows for MCP is only available officially on Mac's and the web interface is generally more convenient for non-technical users. Searching and interacting with the web has become a table-stakes feature and was a glaring gap in Claude.
Let me rephrase it.
MCP has the capability to add this functionality.
It would be nice to see MCP getting adoption in their web UI, as well easier UX, rather than more ad hoc features being added natively.
Re: Claude can now search the web
#43[deleted]
Re: Claude can now search the web
#44Earlier quoted context omitted.
almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off
In what circles is it a joke? Google bots seem to respect it on my sites according to logs.
I know an artist that had noindex turned on by mistake in robots.txt for the last 5 years - google, kagi and duckduckgo find tons of links relevant to the artist and the artwork but not a single one from the website.
so not seem to or apparently but matter of fact like. robots.txt works for the intended audience
Re: Claude can now search the web
#45Aside, does anyone know of an app like Perplexity for surfing the news in a foreign language (language practice)? Perplexity's "Explore" tab translates its news to your local language, and its curated news items are all pretty interesting, but the problem is that there are so few of them. I seem to get maybe a dozen stories in a day. I paid their subscription for a month just to listen to the news on my walk, but did…
Use a VPN to appear like being in the target country.
Use a browser profile where you set the language preference to the one you target.
Re: Claude can now search the web
#46Earlier quoted context omitted.
robots.txt is meant for automated crawlers, not human-driven actions.
It must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.
I remember in late 90s/early 2000 as a teen going to robots.txt to specifically see what they were trying to hide and exploring those urls.
What is the difference if I use a browser or a LLM tool (or curl, or wget, etc) to make those requests?
Re: Claude can now search the web
#47Re: Claude can now search the web
#48Earlier quoted context omitted.
almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off
In what circles is it a joke? Google bots seem to respect it on my sites according to logs.
It's in a small circle of those that do. Blame the internet archive for starting this trend: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Re: Claude can now search the web
#49These are interesting times.
It wasn't long ago that a uni senior who worked for a decade+ on Google Search told me that it was hopeless anyone tries to compete with Google not because it sees a tonne of signals that helps with IR but because of its in-house AI/ML.
It turns out that the org that built the ultimate AI/ML that runs rings around anything that came before it for NLP (and thus IR) was a sister team at Google Translate.
It isn't inconceivable that a kid might be able to build a Google-quality web search, scalability aside, on CommonsCrawls data in a weekend. As someone who built re-ranking algorithms for a search engine built atop Yahoo! and Wikipedia (REST/SOAP) APIs back in the late 2000s as a side project (and experienced the launch and subsequent iterations of Echo/Alexa up close at Amazon), the current capabilities (of even the open weight multi-modal models) seem too good to be true.
Google itself though is saved by its enormous distribution advantages afforded by Chrome (3B to 5B users) and Android (3B+), aside from its search deals with Apple and other browser vendors.
Re: Claude can now search the web
#50I wonder if it will actually respect the robots.txt this time.
robots.txt is meant for automated crawlers, not human-driven actions.
In practice, robots.txt is to control which pages appear in Google results, which is respected as a matter of courtesy, not legality. It doesn't prevent proxies etc. from accessing your sites.