Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I've built a locally running Perplexity clone

github.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Show HN: I've built a locally running Perplexity clone

#111

Post by dns_snek »

openquery wrote: post #85
wg0 wrote: post #82

In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker registry of AI models which is pretty much Hugging Face already there. This all would be due to optimisations within model inference code and techniques, hardware and packaging of software like the above. Don't see billion dollar valuations for lots of AI startups o…

> I foresee that lots of inference would be happening on local machines with models being downloaded on demand Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet. By analogy, you don't have a search engine running on your phone right?

> Why?

Privacy, security, latency, offline availability, access to local data and services running on the device, just to name a few.

Re: Show HN: I've built a locally running Perplexity clone

#112

Post by nilsherzig »

d-z-m wrote: post #81
nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

any plans to support other backends besides ollama?

Sure (if they are openai api compatible i can add them within minutes) otherwise I'm open for pull requests :)

Also, i don't own an Nvidia Card or Windows / MacOS

Re: Show HN: I've built a locally running Perplexity clone

#113

Post by nilsherzig »

madeofpalk wrote: post #108

> Q: is chrome on ios powered by safari > According to the sources provided, Chrome on iOS is not powered by Safari. Google's Chrome uses the Blink engine, while Safari uses the WebKit engine. I find it amusing how when people show off their LLM projects their examples are always of it failing, and providing a bad answer.

Well i don't indent to get money from people, so i guess showing real results isnt a "problem".

Besides i think the following sentences arent wrong? Its just a 7b model give it some slack haha

Re: Show HN: I've built a locally running Perplexity clone

#114

Post by nilsherzig »

ldjkfkdsjnv wrote: post #23

The big secret about perplexity is they havent done much beyond using off the shelf models

I assume the same, feels like their product is just summarizing the top n results? I wouldn't need the whole vector db thing, if local models (or hardware) would be able to run with a context of this size.

Re: Show HN: I've built a locally running Perplexity clone

#116

Post by htrp »

nilsherzig wrote: post #99
sebzim4500 wrote: post #95

Whenever I see these projects I always find reading the prompts fascinating. > Useful for searching through added files and websites. Search for keywords in the text not whole questions, avoid relative words like "yesterday" think about what could be in the text. > The input to this tool will be run against a vector db. The top results will be returned as json. Presumably each clarification is an attempt to fix a bug…

haha yea pretty much, its amazing (and frustrating) how much of the programs "performance" depends on these prompts

Our current state of the art

also love your last commit

>fix: copilot is stupid and i should not blindly trust it

>https://github.com/nilsherzig/LLocalSearch/commit/9f45e24f15...

Everything wrong with code gen in a nutshell

Re: Show HN: I've built a locally running Perplexity clone

#117

Post by madeofpalk »

nilsherzig wrote: post #113
madeofpalk wrote: post #108

> Q: is chrome on ios powered by safari > According to the sources provided, Chrome on iOS is not powered by Safari. Google's Chrome uses the Blink engine, while Safari uses the WebKit engine. I find it amusing how when people show off their LLM projects their examples are always of it failing, and providing a bad answer.

Well i don't indent to get money from people, so i guess showing real results isnt a "problem". Besides i think the following sentences arent wrong? Its just a 7b model give it some slack haha

No, sure. I just think it's funny how it constantly happens, from opensource, free, or commercial projects. No one seems to be immune to 'telling on themselves'.

Re: Show HN: I've built a locally running Perplexity clone

#118

Post by nilsherzig »

htrp wrote: post #116
nilsherzig wrote: post #99

Earlier quoted context omitted.

haha yea pretty much, its amazing (and frustrating) how much of the programs "performance" depends on these prompts

Our current state of the art also love your last commit >fix: copilot is stupid and i should not blindly trust it > https://github.com/nilsherzig/LLocalSearch/commit/9f45e24f15... Everything wrong with code gen in a nutshell

Yea im kinda stressed out to get it working for everyone haha. I would have caught that under different conditions. I'm a big e2e tests guy haha

Re: Show HN: I've built a locally running Perplexity clone

#119

Post by ziziman »

nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

To scrape the websites, do you just blindly cut all of the HTML into defined size chunks or is there some more sophisticated logic to extract text of interest ?

I'm wondering because most news websites now have a lot of polluting elements like popups, would they also go into the database ?

Re: Show HN: I've built a locally running Perplexity clone

#120

Post by totolouis »

ziziman wrote: post #119
nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

To scrape the websites, do you just blindly cut all of the HTML into defined size chunks or is there some more sophisticated logic to extract text of interest ? I'm wondering because most news websites now have a lot of polluting elements like popups, would they also go into the database ?

If you look at the vector handler in his code, he is using blue Monday sanitizer and doing some "replaceAll".

So I think there may be some useless data in the vector, but that may not be a issue since it is coming from multiple sources (for simple question at least)

Post reply on HN