111–120 of 182 posts
I'm looking to use web search in production, but they haven't mentioned the price. Only thing that's mentioned is $20/month, but how much quota does it include?
Dumb question: is this affiliated with Meta? Or is this just someone trying to monetize Meta open source models?
Earlier quoted context omitted.
OK, so it looks like you aren't willing to share which providers you are working with. Can you share the rationale for not sharing that information instead?
We have relationships with many providers and I don't want to be seen as promoting or not promoting a specific provider. Some decent privacy-preserving vendors - Brave, Exa, Parallel Web Systems, DuckDuckGo etc We will continue to monitor what's good to improve the output quality and results. Sometimes it could be the combination of providers to yield even better results. If I say one combination right now, and reali…
I added search to my LLMs years ago with the python DuckDuckGo package. However I found that Google gives better results, so I switched to that. (I forget exactly but I had to set up something in a Google dev console for that.) I think the DDG one is unofficial, and the Google one has limits (so it probably wouldn't work well for deep research type stuff). I mostly just pipe it into LLM apis. I found that "shove the…
How do you meaningfully use it without using scraping APIs? Aren't the official apis severely limited?
Earlier quoted context omitted.
OK, so it looks like you aren't willing to share which providers you are working with. Can you share the rationale for not sharing that information instead?
We have relationships with many providers and I don't want to be seen as promoting or not promoting a specific provider. Some decent privacy-preserving vendors - Brave, Exa, Parallel Web Systems, DuckDuckGo etc We will continue to monitor what's good to improve the output quality and results. Sometimes it could be the combination of providers to yield even better results. If I say one combination right now, and reali…
Brave: https://api-dashboard.search.brave.com/terms-of-service "Licensee shall not at any time, and shall not permit others to: store the results of the API or any derivative works from the results of the API"
Exa: https://exa.ai/assets/Exa_Labs_Terms_of_Service.pdf "You may not [...] download, modify, copy, distribute, transmit, display, perform, reproduce, duplicate, publish, license, create derivative works from, or offer for sale any information contained on, or obtained from or through, the Services, except for temporary files that are automatically cached by your web browser for display purposes"
Many of the things I want to do with a search API are blocked by these rules! So I need to know which rules I am subject to.
On a slightly related note- I've been thinking about building a home-local "mini-Google" that indexes maybe 1,000 websites. In practice, I rarely need more than a handful of sites for my searches, so it seems like overkill to rely on full-scale search engines for my use case. My rough idea for architecture: - Crawler: A lightweight scraper that visits each site periodically. - Indexer: Convert pages into text and cre…
You could take a look at the leaked Yandex source code from a few years ago. I'd believe their architecture should be decent enough.
I'd love to know what search engine provider they're using under the hood for this. I asked them on Twitter and didn't get a reply (yet) https://twitter.com/simonw/status/1971210260015919488 Crucially, I want to understand the license that applies to the search results. Can I store them, can I re-publish them? Different providers have different rules about this.
Earlier quoted context omitted.
We have relationships with many providers and I don't want to be seen as promoting or not promoting a specific provider. Some decent privacy-preserving vendors - Brave, Exa, Parallel Web Systems, DuckDuckGo etc We will continue to monitor what's good to improve the output quality and results. Sometimes it could be the combination of providers to yield even better results. If I say one combination right now, and reali…
The reason I care about this is that different providers have different rules about how I can use the results. Brave: https://api-dashboard.search.brave.com/terms-of-service "Licensee shall not at any time, and shall not permit others to: store the results of the API or any derivative works from the results of the API" Exa: https://exa.ai/assets/Exa_Labs_Terms_of_Service.pdf "You may not [...] download, modify, copy,…
Earlier quoted context omitted.
We have relationships with many providers and I don't want to be seen as promoting or not promoting a specific provider. Some decent privacy-preserving vendors - Brave, Exa, Parallel Web Systems, DuckDuckGo etc We will continue to monitor what's good to improve the output quality and results. Sometimes it could be the combination of providers to yield even better results. If I say one combination right now, and reali…
This information is very useful to the open source community. Whats the rationale in not "building in the public"? Is Ollama turning its back on the open source community? Also why should we believe ollama web search is better than my locally run searxng server?
On making the search functionality locally -- we made considerations and gave it a try but had trouble around result quality and websites blocking Ollama for making a crawler. Using a hosted API, we can get results for users much faster. I'd want us to revisit this at some point. I believe in having the power of local.