91–100 of 196 posts
I think for stuff that isn’t super private like code and such, it’s not worth the effort
Tangential question. What do people use for search? What search engines provide the best quality to cost ratios? Also are there good solutions for searching through a local collection of documents?
There's also google, which gives you 100 requests a day or something.
Here's the search.py I use
import os
import json
from req import get
# https://programmablesearchengine.google.com/controlpanel/create
GOOGLE_SEARCH_API_KEY = os.getenv('GOOGLE_SEARCH_API_KEY')
GOOGLE_SEARCH_API_ID = os.getenv('GOOGLE_SEARCH_API_ID')
url = "https://customsearch.googleapis.com/customsearch/v1"
def search(query):
data = {
"q": query,
"cx": GOOGLE_SEARCH_API_ID,
"key": GOOGLE_SEARCH_API_KEY,
}
results_json = get(url, data)
results = json.loads(results_json)
results = results["items"]
return results
if __name__ == "__main__":
while True:
query = input('query: ')
results = search(query)
print(results)
from duckduckgo_search import DDGS
def search(query, max_results=8):
results = DDGS().text(query, max_results=max_results)
return resultsEarlier quoted context omitted.
gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…
you can run the 120b model on an 8GB GPU? or are you running this on CPU with the 64GB RAM? I'm about to try this out lol The 20b model is not great, so I'm hoping 120b is the golden ticket.
And like a dumbass I accidentally deleted the directory and didn't have a back up or under version control.
Either way, I do know for a fact that the gpt-oss-XXb model beat chatgpt by 1 answer and it was 46/50 at 6 minutes and 47/50 at 1+ hour. I remember because I was blown away that I could get that type of result running locally and I had texted a friend about it.
I was really impressed but disappointed at the huge disparity between time the two.
I sometimes still code with a local LLM but can't imagine doing it on a laptop. I have a server that has GPUs and runs llama.cpp behind llama-swap (letting me switch between models quickly). The best local coding setup I've been able to do so far is using Aider with gpt-oss-120b. I guess you could get a Ryzen AI Max+ with 128GB RAM to try and do that locally but non-nVidia hardware is incredibly slow for coding usage…
gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…
Earlier quoted context omitted.
gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…
you can run the 120b model on an 8GB GPU? or are you running this on CPU with the 64GB RAM? I'm about to try this out lol The 20b model is not great, so I'm hoping 120b is the golden ticket.
Any halo strix laptop, I have been using the hp zbook ultra g1a with 128gb of unified memory. Mostly with the 20B parameters models but it can load larger ones. I find local models (gpt oss 20B) are good quick references but if you want to refactor or something like that you need a bigger model. I’m running llama.cpp directly and using the api it offers for neovim’s avante plugin, or a cli tool like aichat, it comes…
Do you run into hibernation/sleep issues under current mainline Linux kernels by chance? I have this laptop and that's the only thing which isn't working out of the box for me on the Linux side, but it works fine in Windows. I know it's officially supported under the Ubuntu LTS, but I was hoping that wouldn't be needed as I do want a newer+customized kernel.
I got sleep working by disabling webcam in the bios for now.
Earlier quoted context omitted.
I've had good luck with GPT-OSS-120b (reasoning_effort set to "high") + Codex + llama.cpp all running locally, but I needed to do some local patches to Codex as they don't allow configuring and setting the right values for temperature and top_p for GPT-OSS. Also heavy prompting via AGENTS.md was needed to get it to have similar workflow to GPT-5, it didn't seem to pick up that by itself, so I'm assuming GPT-5 been tr…
Would love for you to share the Codex patches you needed to make and the AGENTS.md prompting, if you're open to it.
> Which model(s) are you running (e.g., Ollama, LM Studio, or others) I'm running mainly GPT-OSS-120b/20b depending on the task, Magistral for multimodal stuff and some smaller models I've fine-tuned myself for specific tasks.. All the software is implemented by myself, but I started out with basically calling out to llama.cpp, as it was the simplest and fastest option that let me integrate it into my own software wi…
gpt-oss-120b keeps stopping for me in Codex. (Also in Crush.) I have to say "continue" constantly.
I sometimes still code with a local LLM but can't imagine doing it on a laptop. I have a server that has GPUs and runs llama.cpp behind llama-swap (letting me switch between models quickly). The best local coding setup I've been able to do so far is using Aider with gpt-oss-120b. I guess you could get a Ryzen AI Max+ with 128GB RAM to try and do that locally but non-nVidia hardware is incredibly slow for coding usage…
My current setup is the llama-vscode plugin + llama-server running Qwen/Qwen2.5-Coder-7B-Instruct. It leads to very fast completions, and don't have to worry about internet outages which take me out of the zone.
I do wish qwen-3 released a 7B model supporting FIM tokens. 7B seems to be the sweet spot for fast and usable completions