Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I've built a locally running Perplexity clone

github.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: Show HN: I've built a locally running Perplexity clone

#91

Post by [deleted] »

Sammi wrote: post #88
openquery wrote: post #85

Earlier quoted context omitted.

> I foresee that lots of inference would be happening on local machines with models being downloaded on demand Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet. By analogy, you don't have a search engine running on your phone right?

You currently can't have a search engine running locally on your phone. Google search is possible the single largest c++ program every built. And nevermind the storage needs... But in a few years we might be able to have LLMs running on our phones that work just as well if not better. Of couse as you mention the LLMs running on large servers might still be much more powerfull, but the local ones might be powerfull en…

[deleted]

Re: Show HN: I've built a locally running Perplexity clone

#92

Post by regularfry »

keyle wrote: post #6
viksit wrote: post #5

Earlier quoted context omitted.

curious what hardware you use? and is any of this runnable on an m1 laptop?

Absolutely, 7B will run comfortably on 16GB of RAM and most consumer level hardware. Some of the 40B run on 32GB, but it depends on the model I found (GGUF, crossing fingers help). I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem. I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web. 7b is obviously less capable but the in…

I've been occasionally using a 7b Q4 quant on llama.cpp on an 8GB M1. It's usable, if not amazing.

Re: Show HN: I've built a locally running Perplexity clone

#93

Post by ThinkBeat »

wg0 wrote: post #82

In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker registry of AI models which is pretty much Hugging Face already there. This all would be due to optimisations within model inference code and techniques, hardware and packaging of software like the above. Don't see billion dollar valuations for lots of AI startups o…

"640kb will be enough for everyone." (Gates)

I think that the models will evolve and grow as more powerful compute/hardware comes out.

You may be able to run scaled down n versions of what state of the art now, but by then the giant models will have grown in size and in required compute.

The 6 year old models will be retro computingish.

Somewhat like how you can play 6 year old games on a new powerful PC but by then the new huge games will no longer play well on your Old mach

Re: Show HN: I've built a locally running Perplexity clone

#94

Post by rizky05 »

manojlds wrote: post #50
visarga wrote: post #45

Earlier quoted context omitted.

As a last resort we could have AI work on top of a real web browser and solving captchas as well. Should look like normal usage. I think these kinds of systems LLM + RAG + Web Agent will become widespread and the preferred method to interact with the web. We can escape all ads and dark UI patterns by delegating this task to AI agents. We could have it collect our feeds, filter, rank and summarize them to our preferen…

If everyone consumes like that what's even the incentive for content creators?

[dead]

Re: Show HN: I've built a locally running Perplexity clone

#95

Post by sebzim4500 »

Whenever I see these projects I always find reading the prompts fascinating.

> Useful for searching through added files and websites. Search for keywords in the text not whole questions, avoid relative words like "yesterday" think about what could be in the text. > The input to this tool will be run against a vector db. The top results will be returned as json.

Presumably each clarification is an attempt to fix a bug experienced by the developer, except the fix is in English not in Go.

Re: Show HN: I've built a locally running Perplexity clone

#99

Post by nilsherzig »

sebzim4500 wrote: post #95

Whenever I see these projects I always find reading the prompts fascinating. > Useful for searching through added files and websites. Search for keywords in the text not whole questions, avoid relative words like "yesterday" think about what could be in the text. > The input to this tool will be run against a vector db. The top results will be returned as json. Presumably each clarification is an attempt to fix a bug…

haha yea pretty much, its amazing (and frustrating) how much of the programs "performance" depends on these prompts

Re: Show HN: I've built a locally running Perplexity clone

#100

Post by nilsherzig »

andrewfromx wrote: post #98
andrewfromx wrote: post #97

searXNGDomain := os.Getenv("SEARXNG_DOMAIN") I see this but what search engine lets you get results in json for free?

ohh " https://duckduckgo.com/?q=andrew&format=json " nice!

Those arent search results tho, that's just duckduckgo internal things like "similar queries"
Post reply on HN