Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude can now search the web

anthropic.com

Post reply on HN
Print view View original

541–550 of 758 posts

Re: Claude can now search the web

#541

Post by hk__2 »

gizmodo59 wrote: post #406
ordersofmag wrote: post #359

Earlier quoted context omitted.

In my quick experiment (asking a question that would naturally lead to content on my own site) it is not doing a real time request to the site in question. Its answer included links back to my site (and relevant summaries), but there was no requests for those pages while it was generating its answer. So it's clearly drawing from info that has already been scraped at some earlier point. And given that I see Claudebot…

Major AI players don’t want to use someone else web index as they may cut it off or jack up the prices etc. major players want to build their own web index

And this is why we see our logs overloaded with ABot BBot CBot etc, every single "AI" company makes their own bot and they all crawl the same pages over and over.

Re: Claude can now search the web

#542

Post by Sharlin »

shortrounddev2 wrote: post #419
herdcall wrote: post #362

It badly hallucinated in my test. I asked it "Rust crate to access Postgres with Arrow support" and it made up an arrow-postgres crate. It even gave sample Rust code using this fictional crate! Below is its response (code example omitted): I can recommend a Rust crate for accessing PostgreSQL with Arrow support. The primary crate you'll want to use is arrow-postgres, which combines the PostgreSQL connectivity of the…

> I asked it "Rust crate to access Postgres with Arrow support" Is that how you actually use llms? Like a Google search box?

It's absolutely how LLMs should work, and IME they do. Why write a full question if a search phrase works just as well? Everything in "Could you recommend xyz to me?" except "xyz" is redundant and only useful when you talk to actual humans with actual social norms to observe. (Sure, there used to be a time when LLMs would give better answers if you were polite to them, but I doubt that matters anymore.) Indeed I've been thinking of codifying this by adding a system prompt that says something like "If the user makes a query that looks like a search phrase, phrase your response non-conversationally as well".

Re: Claude can now search the web

#543

Post by JimDabell »

manquer wrote: post #418
JimDabell wrote: post #231

Earlier quoted context omitted.

The LLM shouldn’t. robots.txt is intended to control recursive fetches. It is not intended to block any and all access. You can test this out using wget. Fetch a URL with wget. You will see that it only fetches that URL. Now pass it the --recursive flag. It will now fetch that URL, parse the links, fetch robots.txt, then fetch the permitted links. And so on. wget respects robots.txt. But it doesn’t even bother lookin…

While robots.txt is not there to directly prevent automated requests, it does prevent crawling which is needed for search indices. Without recursive crawling, it will not possible for a engine to know what are valid urls[1]. They will otherwise either have to brute-force say HEAD calls for all/common string combinations and see if they return 404s or more realistically have to crawl the site to "discover" pages. The…

> Robots.txt does absolutely apply to LLMs engines and search engines equally.

It does not. It applies to whatever crawler built the search index the LLM accesses, and it would apply to an AI agent using an LLM to work recursively, but it does not apply to the LLM itself or the feature being discussed here.

The rest of your comment seems to just be repeating what I already said:

> Whatever search index they are using, the crawler for that search index needs to respect robots.txt because it’s acting recursively. But when the user asks the LLM to look at web results, it’s just getting a single set of URLs from that index and fetching them – assuming it’s even doing that and not using a cached version. It’s not acting recursively, so robots.txt does not apply.

There is a difference between an LLM, an index that it consults, and the crawler that builds that index, and I was drawing that distinction. You can’t just lump an LLM into the same category, because it’s doing a different thing.

Re: Claude can now search the web

#544

Post by LightBug1 »

Almondsetat wrote: post #111
tcdent wrote: post #102

Searching the web is a great feature in theory, but every implementation I've used so far looks at the top X hits and then interprets it to be the correct answer. When you're talking to an LLM about popular topics or common errors, the top results are often just blogspam or unresolved forum posts, so the you never get an answer to your problem. More of an indicator that web search is more unusable than ever, but inte…

>looks at the top X hits and then interprets it to be the correct answer. LLMs are truly reaching human-like behavior then

Degenerative AI ?

Re: Claude can now search the web

#545

Post by bluebarbet »

blackqueeriroh wrote: post #493
rank0 wrote: post #431

Earlier quoted context omitted.

Right! So everyone would choose to pursue passions/interests/leisure. We would be going into debt with no meaningful benefit to the taxpayer. Direct malinvestment.

It seems to me that you’re confused about what people enjoy doing. Also, it’s fascinating that you say “no benefit to the taxpayer” as if the taxpayer not having to work is somehow not a benefit?

>It seems to me that you’re confused

A conversation that starts like this is not going to go well.

Re: Claude can now search the web

#546

Post by dmazin »

HankWozHere wrote: post #124
tcdent wrote: post #102

Searching the web is a great feature in theory, but every implementation I've used so far looks at the top X hits and then interprets it to be the correct answer. When you're talking to an LLM about popular topics or common errors, the top results are often just blogspam or unresolved forum posts, so the you never get an answer to your problem. More of an indicator that web search is more unusable than ever, but inte…

Kagi Assistant allows you to do search with LLM queries. So far I feel it bears reliable results. For instance - I tried couple of queries for product suggestions and came back with some good results. Whilst it’s a premium service , I find the offering to be of good value.

Has anyone compared Perplexity with Kagi Assistant?

I am always looking for Perplexity alternatives. I already pay for Kagi and would be happy to upgrade to the ultimate plan if it truly can replace Perplexity.

Re: Claude can now search the web

#548

Post by computerthings »

sshine wrote: post #466
labrador wrote: post #440

Earlier quoted context omitted.

Thank you for saying this. I recently heard that story "if you let one Nazi in your bar, pretty soon you have a Nazi bar"

Also, if you call everyone a nazi, all you have is nazi bars. I was called a nazi last week for driving a Tesla, and I have Jewish ancestry. The word hardly makes any sense.

[dead]

Re: Claude can now search the web

#549

Post by jascha_eng »

sshine wrote: post #489
wongarsu wrote: post #308

Earlier quoted context omitted.

Purely on its technical merits Grok is pretty good and fills a niche in the selection of AI agents. But I can absolutely understand not wanting to use an AI owned by somebody who makes Nazi salutes and is dismantling the US government.

[flagged]

As a German: He did, and even if there is a chance that he didn't mean it like that the risk of another 1933 is not worth it. I usually don't like cancel-culture but you have to have boundaries and I think the risk of another Holocaust and all the other Nazi cruelties is a boundary a functioning society should be able to agree on.

Re: Claude can now search the web

#550

Post by computerthings »

sshine wrote: post #487
appleaday1 wrote: post #480

Earlier quoted context omitted.

[flagged]

I don’t respond well to peer pressure. It makes me sick to the stomach. Peer pressure from aggressive behaviour is ironically how Germany’s population got talked into committing genocide. I’ll start doing what other people say for no good reason the day I switch off my brain.

[dead]
Post reply on HN