1–10 of 140 posts
Show HN: I've built a locally running Perplexity clone
github.com
I too use local 7b open-hermes and it's really good.
It's basically a LLMs with access to a search engine and the ability to query a vector db.
The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM just summarize everything, but at least on local hardware it's a lot faster and way more resource friendly. The demo on GitHub runs on a normal consumer GPU (amd rx 6700xt) with 12gb vRAM.
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.
curious what hardware you use? and is any of this runnable on an m1 laptop?
I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem.
I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web.
7b is obviously less capable but the instant response makes it worth exploring. It's very useful as a Google search replacement that is instantly more valuable, for general questions, than dealing with the hellscape of blog spam ruling Google atm.
Note, for my complex code queries at $dayjob where time is of the essence, I still use GPT4 plus, which is still unmatched imho, without running special hardware at least.
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.
Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.
curious what hardware you use? and is any of this runnable on an m1 laptop?
Earlier quoted context omitted.
Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.
Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!
“Needs tool usage” and “found the answer” blocks in your infra, how are these decisions made?
Looking at the demo, it takes a little time to return results, from the search, vector storage and vector db retrieval, which step takes the most time?