Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I've built a locally running Perplexity clone

github.com

Post reply on HN
Print view View original

61–70 of 140 posts

Re: Show HN: I've built a locally running Perplexity clone

#61

Post by nilsherzig »

ivolimmen wrote: post #34
nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

"normal consumer GPU"... well mine is a 4GB 6600.. so I guess that varies.

Sorry it wasn't my intention to gatekeep, but my 300€ card really is on the low end of LLM Things

Re: Show HN: I've built a locally running Perplexity clone

#62

Post by nilsherzig »

mark_l_watson wrote: post #26
nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

Your project looks very cool. I had on my ‘list’ to re-learn Typescript (I took a TS course about 5 years ago, but didn’t do anything with it) so I just cloned your repo so I can experiment with it. EDIT: I just noticed that most of the code is Go. Still going to play with it!

Thanks :). Yea only the web part is typescript and I really wouldn't recommend to learn from my typescript haha

Re: Show HN: I've built a locally running Perplexity clone

#63

Post by nilsherzig »

mmahemoff wrote: post #37
nilsherzig wrote: post #19

Earlier quoted context omitted.

searxng, which is a locally running meta search engine combining a lot of different sources (including Google and co)

This might be more of a searxng question, but doesn't it quickly run up against anti-bot measures? CAPTCHA challenges and Forbidden responses? I can see the manual has some support for dealing with CAPTCHA [1], but in practical terms, I would guess a tool like this can't be used extensively all day long. I'm wondering if there's a search API that would make the backend seamless for something like this. 1. https://doc…

I didn't run into a lot of timeouts while using it myself, but you would probably need another search source if you plan to host this service for multiple users at the same time.

There are projects like flareresolverr which might be interesting

Re: Show HN: I've built a locally running Perplexity clone

#64

Post by nilsherzig »

arflikedog wrote: post #48

A while back you commented on my personal project Airdraw which I really appreciated. This looks awesome and you're well on your way to another banger project - looking forward to toying around with this :)

Uhh yes I was really impressed by your project :)

Re: Show HN: I've built a locally running Perplexity clone

#66

Post by nilsherzig »

noisy_boy wrote: post #57

Would be good if the readme mentions minimum hardware specs to get a reasonably decent performance. E.g. I have a ThinkPad X1 extreme i7 with MaxQ graphics, any hopes of running this on it without completely ruining the performance?

You could run the LLM using your CPU and normal (non video) ram. But that's a lot slower. There are people working on making it a lot faster tho. The bottleneck is the transfer speed between the ram Sticks and the CPU.

Just taking a guess, but I wouldn't expect more than a couple tokens (more or less like syllables) per second. Which is probably to slow, since it has to read a couple thousand per search result.

It's hard to provide minimum requirements, since there are so many edge cases.

Re: Show HN: I've built a locally running Perplexity clone

#67

Post by nilsherzig »

hubraumhugo wrote: post #46

Excellent work! Cool side projects like that will eventually help you get hired by a top startup or may even lead to building your own. I can only encourage other makers to post their projects on HN and put them out into the world.

Yea it's also quite fulfilling to see people likening something you've put some work into :)

Re: Show HN: I've built a locally running Perplexity clone

#70

Post by lavela »

nilsherzig wrote: post #15
pants2 wrote: post #12

It says it's a "locally running search engine" - but not sure how it finds the sites and pages to index in the first place?

Yea I guess that's misleading, I should probably change that. I was referring to the LLM part as locally running. Indexing is still done by the big guys and queried using searxng

What would be your current recommendation on how to create a vector db from local files that would work with LLocalSearch?
Post reply on HN