151–160 of 182 posts
Looks like Ollama is focusing more and more on non-local offerings. Also their performance is worse than say vLLM. What's a good Ollama alternative (for keeping 1-5x RTX 3090 busy) if you want to run things like open-webui (via an OpenAI compatible API) where your users can choose between a few LLMs?
Are the rate limits documented somewhere?
WTF is going to happen to Google's ad revenue if every PC has an AI that can perform searches on the user's behalf?
Earlier quoted context omitted.
How do you meaningfully use it without using scraping APIs? Aren't the official apis severely limited?
Google Programmable Search Engine [1] is pretty good if your needs are within their usage limits. [1] https://programmablesearchengine.google.com/about/
https://programmablesearchengine.google.com/controlpanel/cre...
And then it's just a GET:
import os
import json
from req import get
url = "https://customsearch.googleapis.com/customsearch/v1"
def search(query):
data = {
"q": query,
"cx": os.getenv('GOOGLE_SEARCH_API_KEY'),
"key": os.getenv('GOOGLE_SEARCH_API_ID')
}
results_json = get(url, data)
results = json.loads(results_json)
results = results["items"]
return resultsEarlier quoted context omitted.
There are very few recently launched pure open source projects these days (most are at least running donation-ware models or funded by corporate backers), none in the AI space that I'm aware of.
Well the real open source project is llama.cpp which Ollama basically wrapped and made a nice interface on top of. Now they do more things as they want to be a real business, but llama.cpp is now doing most things people wanted from something like ollama, like serving a REST API compatible with OpenAPI, downloading and managing local LLMs… while remaining an actual open source project without VC money as far as I kno…
This is a new umbrella project for llama.cpp and whisper.cpp. The author, Georgi Gerganov, also announced he’s forming a company for the project as he raised money from Nat Friedman (CEO GitHub) and Daniel Gross (ex-YC AI, ex-Apple ML).
Not sure if this is just a good faith support.
Earlier quoted context omitted.
What sort of monetization model would you like to see? What model would you deem acceptable?
Ollama , the local inference platform, stays completely local. Maintained by a non-profit org with dev time contributed to by a for-profit company. That company can be VC backed and can make their cloud inference platform. And can use ollama as its backed, as a platform to market etc. But keep it as a separate product (not named ollama). This is almost exactly how duckdb/motherduck functions and I think theyre doing…
I tried it a while back, I was very surprised to find that simply running `uvx ramalama run deepseek-r1:1.5b` just worked. I'm on Fedora Silverblue with nothing layered on the ostree. Before RamaLama, getting llama.cpp working with my GPU was a major PITA.
Earlier quoted context omitted.
More competition in the space would be great for me as a consumer, but the problem is that the high fixed costs make starting an index difficult.
I've been wondering can't this be done p2p? Didn't we solve most of the technical problems in the late 90s / early 2000s? And then just abandoned that entire way of thinking for some reason? If many thousands of people care about having a free / private / distributed search engine, wouldn't it make sense for them to donate 1% of their CPU/storage/network to an indexer / db that they they then all benefit from?
How do you make it trustless. How do you fetch/crawl the index when it's scattered across arbitrary devices. How do you index the decentralized index. What is actually stored on nodes. When you want to do something useful with the crawled info, what does that look like.
WTF is going to happen to Google's ad revenue if every PC has an AI that can perform searches on the user's behalf?
Earlier quoted context omitted.
We have relationships with many providers and I don't want to be seen as promoting or not promoting a specific provider. Some decent privacy-preserving vendors - Brave, Exa, Parallel Web Systems, DuckDuckGo etc We will continue to monitor what's good to improve the output quality and results. Sometimes it could be the combination of providers to yield even better results. If I say one combination right now, and reali…
The reason I care about this is that different providers have different rules about how I can use the results. Brave: https://api-dashboard.search.brave.com/terms-of-service "Licensee shall not at any time, and shall not permit others to: store the results of the API or any derivative works from the results of the API" Exa: https://exa.ai/assets/Exa_Labs_Terms_of_Service.pdf "You may not [...] download, modify, copy,…
Earlier quoted context omitted.
Ya a lot of ppl don't realize you could spend 2k on a 5090 to run some of the large models. Or spend 20 a month for models even a 5090 couldn't run. And not have to spend your own electricity, hardware, maintenance, updates etc.
20 a month for a commercial model is price dumping financed by investors. For ollama it's hopefully a sustainable price.
And with that in mind, i definetly dont use more than a couple of bucks a month in API refils. (not that i really am a power user or anything)
So if you consider the 20 bucks to be balanced between poer and non power users, and with the existing rate limits, its probably not that far off being profitable, at least on the pure inference side.