I wonder if it will actually respect the robots.txt this time.
Why wonder. You can test for yourself.
Claude can now search the web
171–180 of 758 posts
Re: Claude can now search the web
#172Is there a way to access the new web browsing capability via API?
Not yet, and no hints as to whether that will happen or not.
Re: Claude can now search the web
#173Earlier quoted context omitted.
almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off
I have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.
I do essentially both: robots.txt backed by actual server-level enforcement of the rules in robots.txt. You'd think there would be zero hits on the server-level blocking since crawlers are supposed to read and respect robots.txt, but unsurprisingly they don't always. I don't know why this isn't a standard feature in web hosting.
Re: Claude can now search the web
#174Couldn't a lot of front-ends using Claude API do this already? What's new?
This is a new product feature for https://claude.ai and the Claude mobile and desktop apps.
Re: Claude can now search the web
#175Earlier quoted context omitted.
Google search is crap. It seems to be a sentiment among many HNers, but is it really that bad? I mostly use it for programming, so documentation/forums and it works out greatly. For some queries it even returns personal blogs (which people seem to bash google for not happening). Of course there are some queries that return purely AI blogspam, but reformatting the query with a bit more thought usually solves it. I won…
How long have you been using Google search for? It used to be SO much less likely to return junk.
Since around 2012. What year would be the golden age of google search? I wonder if anyone has archived search result pages for relatively timeless queries so that we could compare. Wayback Machine seems to archive some of them.
https://web.archive.org/web/20200801000000*/https://www.goog...
https://web.archive.org/web/20200801000000*/https://www.goog...
Re: Claude can now search the web
#176Earlier quoted context omitted.
ChatGPT can search the web (I just checked in my account). It appears to be available for all users (not just paid): https://help.openai.com/en/articles/9237897-chatgpt-search
I definitely tried to web search with ChatGPT a few weeks ago and it couldn't. I don't think I'm making this up. Unless I suffered a TBI.
Web search is available for some but not all of their models. It is not particularly obvious from their UI which models have this feature.
Re: Claude can now search the web
#177Funny, I literally just two days ago asked Claude to provide an outline of the functionality of a product, giving it the web site. It of course refused. So I downloaded the text of the site and passed that in, and got mediocre results.
The results based on giving the source URL directly were better. Still a bit generic and high-level and vague, as LLMs tend to be, but better than the text-download version a couple days ago. And of course much easier to generate!
Re: Claude can now search the web
#178Great, now we just need a decent search engine.
Re: Claude can now search the web
#179Earlier quoted context omitted.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
No thank you, when I define a robots.txt file I expect all automated systems to respect it.
Any AI tool I make will ignore robots.txt on principle. Artificial humans should have equal rights as real humans.
Re: Claude can now search the web
#180Earlier quoted context omitted.
Is google search bad? Click here to find ten reasons why it is bad and 10 reasons why you should still use it. Yes, it is that bad. Website of Nike? Website of Starbucks? Likely position number one. Every product, category etc., e.g. what rice cooker should I buy? Is diseased by link and affiliate spam. There is a reason why people put +reddit on search terms.
What kind of results does Kagi give for "What rice cooker should I buy?"? (Not asking you specifically, but if any Kagi user could compare.)
Just tried plain old Kagi search, it came up with cooks illustrated (good source, paid) and consumer reports (decent source, paid), which I was surprised by until I remembered that I had these “pinned”, which means Kagi increases their rank. Third on the page was a condensed roundup of 8 listicles, 2 of which seemed decent (food and wine and some random blogger).
With no pins, bon appetit (decent) and nbc news (would be fine if it wasn’t littered with ads) were the top results. For NBC news, Kagi also marked the result with a red shield, indicating that it has too many ads/trackers.
Which really goes to show that Kagi is great if you’re really willing to shell out for better content. Having the ability to mark sources as trusted, or indicate that I’ve paid for premium sources makes a completely different side of the web searchable.