Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I've built a locally running Perplexity clone

github.com

Post reply on HN
Print view View original

121–130 of 140 posts

Re: Show HN: I've built a locally running Perplexity clone

#121

Post by ilc »

dns_snek wrote: post #111
openquery wrote: post #85

Earlier quoted context omitted.

> I foresee that lots of inference would be happening on local machines with models being downloaded on demand Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet. By analogy, you don't have a search engine running on your phone right?

> Why? Privacy, security, latency, offline availability, access to local data and services running on the device, just to name a few.

Big Tech + Countries: Those all sound like great reasons to centralize all access to AIs!

Re: Show HN: I've built a locally running Perplexity clone

#122

Post by lobocinza »

ThinkBeat wrote: post #93
wg0 wrote: post #82

In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker registry of AI models which is pretty much Hugging Face already there. This all would be due to optimisations within model inference code and techniques, hardware and packaging of software like the above. Don't see billion dollar valuations for lots of AI startups o…

"640kb will be enough for everyone." (Gates) I think that the models will evolve and grow as more powerful compute/hardware comes out. You may be able to run scaled down n versions of what state of the art now, but by then the giant models will have grown in size and in required compute. The 6 year old models will be retro computingish. Somewhat like how you can play 6 year old games on a new powerful PC but by then…

There will be demand and supply for both cases.

Re: Show HN: I've built a locally running Perplexity clone

#123

Post by hanniabu »

nilsherzig wrote: post #3

Happy to answer any questions and open for suggestions :) It's basically a LLMs with access to a search engine and the ability to query a vector db. The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM ju…

This is awesome, would love if there were executable files where these dependencies are needed. That would make it wayyyy more accessible rather than just to those that know how to use the command line and resolve dependencies (yes, even docker runs into that when fighting the local system).

Re: Show HN: I've built a locally running Perplexity clone

#124

Post by arflikedog »

gardenhedge wrote: post #89
arflikedog wrote: post #48

A while back you commented on my personal project Airdraw which I really appreciated. This looks awesome and you're well on your way to another banger project - looking forward to toying around with this :)

Did you just happened to see this post today and notice the username?

unironically yes, I used comments to hot fix a bunch of stuff when I first launched. It's a small world and I thought this was a cool moment

Re: Show HN: I've built a locally running Perplexity clone

#125

Post by sroussey »

BrutalCoding wrote: post #84

That’s a great project you pulled off. From the time I starred it (10-12h ago I think), and upon re-checking this post, you gained 500+ stars lol. Visualized in a chart with star-history: https://star-history.com/#nilsherzig/LLocalSearch

Ah, that’s a nice chart generator. Will have to use if I ever get any, lol.

Re: Show HN: I've built a locally running Perplexity clone

#126

Post by peter_l_downs »

BOOSTERHIDROGEN wrote: post #41
peter_l_downs wrote: post #22

Earlier quoted context omitted.

Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-... If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledg…

It's the best balance if you have limited compute performance.

Thank you

Re: Show HN: I've built a locally running Perplexity clone

#127

Post by peter_l_downs »

d-z-m wrote: post #73
peter_l_downs wrote: post #22

Earlier quoted context omitted.

Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-... If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledg…

Talking GGUF, Usually the higher you can afford to go wrt. quantization(e.g. Q5 is better than Q4, etc), the better. A Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that. TheBloke's READMEs[0] usually have a good table summarizing each quantization level. If you're RAM constrained, you'll also have to make trade-offs about the context length. e.g. y…

Thank you!

Re: Show HN: I've built a locally running Perplexity clone

#128

Post by ProllyInfamous »

nilsherzig wrote: post #68
ml-anon wrote: post #47

Did you really make a perplexity clone if you didn’t spend more time promoting yourself on Twitter and LinkedIn than on the engineering?

Ah damn I forgot about getting some VC money

This demo will land you more important things than "just" VC money.

Extremely impressive, cannot wait to actually implement this on my M2Pro (mac).

Re: Show HN: I've built a locally running Perplexity clone

#129

Post by aagha »

According to Crunchbase [0], Perplexity has raised over $100M.

You built this in your spare time?

The following things jump out to me:

- How much a hype cycle invites insane amounts of money - How trash the entire VC world is during a hype cycle - What an amazing thing ingenuity and passion are

Great job!

0 - https://www.crunchbase.com/organization/perplexity-ai

Post reply on HN