Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I've built a locally running Perplexity clone

github.com

Post reply on HN
Print view View original

11–20 of 140 posts

Re: Show HN: I've built a locally running Perplexity clone

#11

Post by nilsherzig »

fnetisma wrote: post #10

This is really neat! I have questions: “Needs tool usage” and “found the answer” blocks in your infra, how are these decisions made? Looking at the demo, it takes a little time to return results, from the search, vector storage and vector db retrieval, which step takes the most time?

Thanks :)

Die LLM makes these decisions on its own. If it writes a message which contains a tool call (Action: Web search Action Input: weight of a llama) the matching function will be executed and the response returned to the LLM. It's basically chatting with the tool.

You can toggle the log viewer on the top right, to get more detail on what it's doing and what is taking time. Timing depends on multiple things: - the size of the top n articles (generating embeddings for them takes some time) - the amount of matching vector DB responses (reading them takes some time)

Re: Show HN: I've built a locally running Perplexity clone

#13

Post by davidcollantes »

nilsherzig wrote: post #4
keyle wrote: post #2

Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.

Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.

Got a link for that one? I have found a few with Hermes-2-Mistral in the name.

Re: Show HN: I've built a locally running Perplexity clone

#14

Post by peter_l_downs »

keyle wrote: post #2

Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.

I'm just starting to get into downloading and testing models using llama.cpp and I'm curious which model you're actually using, since they seem to come in varying levels of quantization. Is this [0] the model page for the one you're using, or should I be looking somewhere else? What is the actual file name of the model you're using?

[0] https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GG...

Re: Show HN: I've built a locally running Perplexity clone

#15

Post by nilsherzig »

pants2 wrote: post #12

It says it's a "locally running search engine" - but not sure how it finds the sites and pages to index in the first place?

Yea I guess that's misleading, I should probably change that. I was referring to the LLM part as locally running. Indexing is still done by the big guys and queried using searxng

Re: Show HN: I've built a locally running Perplexity clone

#16

Post by peter_l_downs »

nilsherzig wrote: post #9
keyle wrote: post #7

Earlier quoted context omitted.

Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!

It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.

Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.

Re: Show HN: I've built a locally running Perplexity clone

#17

Post by koeng »

nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

What is the search engine that it uses?

Re: Show HN: I've built a locally running Perplexity clone

#18

Post by nilsherzig »

nilsherzig wrote: post #15
pants2 wrote: post #12

It says it's a "locally running search engine" - but not sure how it finds the sites and pages to index in the first place?

Yea I guess that's misleading, I should probably change that. I was referring to the LLM part as locally running. Indexing is still done by the big guys and queried using searxng

Just to clarify, it wasn't my intention to be misleading

Re: Show HN: I've built a locally running Perplexity clone

#19

Post by nilsherzig »

koeng wrote: post #17
nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

What is the search engine that it uses?

searxng, which is a locally running meta search engine combining a lot of different sources (including Google and co)

Re: Show HN: I've built a locally running Perplexity clone

#20

Post by BOOSTERHIDROGEN »

peter_l_downs wrote: post #16
nilsherzig wrote: post #9

Earlier quoted context omitted.

It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.

Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.

It's from nous research https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B

Q5 is minimum.

Post reply on HN