Web search on the Anthropic API • Hacker News Board

Re: Web search on the Anthropic API

#41

Post by RainbowcityKun » Thu, May 08, 2025, 2:40 AM UTC

zhyder wrote: post #17

Now all the big 3 LLM providers provide web search grounding in their APIs, but how do they compare in ranking quality of the retrieved web search results? Anyone run benchmarks here? Clearly web search ranking is hard after decades of content spam that's been SEO optimized (and we get to look forward to increasing AI spam dominating the web in the future). The best LLM provider in the future could be the one with ju…

Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited.

The LLMs can access the web, but they can't yet understand it in a structured, evaluative way.

What’s missing is a layer of engineered relevance modeling, capable of filtering not just based on keywords or citations, but on deeper truth alignment and human utility.

And yes, as you mentioned, we may even see the rise of LLM-targeted SEO—content optimized not for human readers, but to game LLM attention and summarization heuristics. That's a whole new arms race.

The next leap won’t be about just accessing more data, but about curating and interpreting it meaningfully.

Re: Web search on the Anthropic API

#42

Post by QuadmasterXLII » Thu, May 08, 2025, 2:56 AM UTC

formercoder wrote: post #37

jwr wrote: post #33

Earlier quoted context omitted.

> Google's pricing for a general search API As I discovered recently, and much to my surprise, Google does not offer a "general search API", at least not officially. There is a "custom search" API that sounds like web search, but isn't: it offers a subset of the index, which is not immediately apparent. Confusing and misleading labeling there. Bing offers something a bit better, but I recently ended up trying the Kag…

I work at Google but not on this. We do offer Gemini with Google Search grounding which is similar to a search API.

??????

Re: Web search on the Anthropic API

#43

Post by potlee » Thu, May 08, 2025, 3:20 AM UTC

stephpang wrote: post #40

potlee wrote: post #15

If you use your own search tool, you would have to pay for input tokens again every time the model decides to search. This would be a big discount if they only charging once for all output as output tokens but seems unclear from the blog post

Thanks for the feedback, just updated our docs to hopefully make this a little clearer. Search results count towards input tokens on every subsequent iteration https://docs.anthropic.com/en/docs/build-with-claude/tool-us...

Thanks for addressing it. Still sounds like a significant discount if only the search results and not all messages count are input tokens on subsequent iterations!

Re: Web search on the Anthropic API

#44

Post by jsnell » Thu, May 08, 2025, 3:35 AM UTC

jwr wrote: post #33

jsnell wrote: post #21

Earlier quoted context omitted.

Yes. The Bing Search API is priced at $15/1k queries in the cheapest tier, Brave API is $9 at the non-toy tier, Google's pricing for a general search API is unknown but their Search grounding in Gemini costs $35/1k queries. Search API prices have been going up, not down, over time. The opposite of LLMs, which have gotten 1000x cheaper over the last two years.

> Google's pricing for a general search API As I discovered recently, and much to my surprise, Google does not offer a "general search API", at least not officially. There is a "custom search" API that sounds like web search, but isn't: it offers a subset of the index, which is not immediately apparent. Confusing and misleading labeling there. Bing offers something a bit better, but I recently ended up trying the Kag…

There are multiple search engines known to be based on Google's API (Startpage, Leta, Kagi), so that product definitely exists. But it exciting that's all we know. They indeed do not publish anything about it. We don't know the price, the terms, or even the name.

Re: Web search on the Anthropic API

#45

Post by metalrain » Thu, May 08, 2025, 4:11 AM UTC

It's a good reminder that AI chats won't make web searches obsolete, just embed them at deeper in the stack.

Maybe Google search revenue moves from ads to more towards B2B deals for search API use.

Re: Web search on the Anthropic API

#46

Post by simianwords » Thu, May 08, 2025, 4:51 AM UTC

Can any one answer this question: are they using custom home made web index? Or are they using bing/google api?

Also I'm quite sure that they don't use vector embeddings for web search, its purely on text space. I think the same holds for all LLM web search tools. They all seem to work well -- maybe we don't need embeddings for RAG and grepping works well enough?

Re: Web search on the Anthropic API

#47

Post by simianwords » Thu, May 08, 2025, 4:53 AM UTC

RainbowcityKun wrote: post #41

zhyder wrote: post #17

Now all the big 3 LLM providers provide web search grounding in their APIs, but how do they compare in ranking quality of the retrieved web search results? Anyone run benchmarks here? Clearly web search ranking is hard after decades of content spam that's been SEO optimized (and we get to look forward to increasing AI spam dominating the web in the future). The best LLM provider in the future could be the one with ju…

Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. The LLMs can access the web, but they can't yet understand it in a structured, evaluative way. What’s missing is a layer of engineered relevance modeling, capable of filtering not just based on keywords or citations, but on de…

>Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited.

Why do you think it is limited? Imagine you show a link with details to an LLM and ask it if it is trustworthy or high quality w.r.t the query, why can't it answer it?

Re: Web search on the Anthropic API

#48

Post by ColinHayhurst » Thu, May 08, 2025, 6:31 AM UTC

jsnell wrote: post #21

jarbus wrote: post #16

Is search really that costly to run? $10/1000 searches seems really pricey. I'm wondering if these costs will come down in a few years.

Yes. The Bing Search API is priced at $15/1k queries in the cheapest tier, Brave API is $9 at the non-toy tier, Google's pricing for a general search API is unknown but their Search grounding in Gemini costs $35/1k queries. Search API prices have been going up, not down, over time. The opposite of LLMs, which have gotten 1000x cheaper over the last two years.

Excuse the self-promotion but Mojeek is £3/1,000: https://www.mojeek.com/services/search/web-search-api/

Re: Web search on the Anthropic API

#49

Post by RainbowcityKun » Thu, May 08, 2025, 6:44 AM UTC

simianwords wrote: post #47

RainbowcityKun wrote: post #41

Earlier quoted context omitted.

Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. The LLMs can access the web, but they can't yet understand it in a structured, evaluative way. What’s missing is a layer of engineered relevance modeling, capable of filtering not just based on keywords or citations, but on de…

>Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. Why do you think it is limited? Imagine you show a link with details to an LLM and ask it if it is trustworthy or high quality w.r.t the query, why can't it answer it?

What I mean is that more powerful engineering capabilities are needed to provide LLM with processing of search results.

Re: Web search on the Anthropic API

#50

Post by simianwords » Thu, May 08, 2025, 6:53 AM UTC

RainbowcityKun wrote: post #49

simianwords wrote: post #47

Earlier quoted context omitted.

>Right now, most LLMs with web search grounding are still in Stage 1: they can retrieve content, but their ability to assess quality, trustworthiness, and semantic ranking is still very limited. Why do you think it is limited? Imagine you show a link with details to an LLM and ask it if it is trustworthy or high quality w.r.t the query, why can't it answer it?

What I mean is that more powerful engineering capabilities are needed to provide LLM with processing of search results.

Not sure I understand -- LLM's are pretty good at assessing quality of search results. If an LLM can bulk assess a bunch of results it can get a pretty far, probably more efficient than a human hand checking all the results.