Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made an app to use local AI as daily driver

recurse.chat

Post reply on HN
Print view View original

171–180 of 246 posts

Re: Show HN: I made an app to use local AI as daily driver

#171

Post by loudmax »

tveita wrote: post #157
danielovichdk wrote: post #149

Earlier quoted context omitted.

I dont get it. To my understanding it takes huge amounts of data to build any any form of RAG. Simply because it enlarges the statistical model you later prompt. If the model is not big enough how would you expect it to answer you in a non qualifying matter ? It simply can't. So I don't really buy it and I have yet to see it work better than any rdbms search index. Tell me I am wrong, I would like to see a local mode…

RAG doesn't require much data or involve any training, it is a fancy name for "automatically paste some relevant context into the prompt" Basically if you have a database of three emails and ask when Biff wanted to meet for lunch, a RAG system would select the most relevant email based on any kind of search - embeddings are most fashionable, and create a prompt like """Given this document: , answer the question "When…

That's not how RAG works. What you're describing is something closer to prompt optimization.

Sibling comment from discordance has a more accurate description of RAG. There's a longer description from Nvidia here: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...

Re: Show HN: I made an app to use local AI as daily driver

#172

Post by castles »

woadwarrior01 wrote: post #166
castles wrote: post #160

Earlier quoted context omitted.

Interesting! What's the prompt eval processing speed like compared to llama.cpp and kin?

I haven't run any specific low level benchmarks, lately. But chunked prefilling and tvm auto-tuned Metal kernels from mlc-llm seemed to make a big differenced, the last time I checked. Also, compared to stock mlc-llm, I use a newer version of metal (3.0) and have a few modifications to make models have a slightly smaller memory and disk footprint, also slightly faster execution. Because unlike the mlc-llm folks, I on…

thanks, I'll give it a crack

Re: Show HN: I made an app to use local AI as daily driver

#175

Post by bigfudge »

discordance wrote: post #162
danielovichdk wrote: post #149

Earlier quoted context omitted.

I dont get it. To my understanding it takes huge amounts of data to build any any form of RAG. Simply because it enlarges the statistical model you later prompt. If the model is not big enough how would you expect it to answer you in a non qualifying matter ? It simply can't. So I don't really buy it and I have yet to see it work better than any rdbms search index. Tell me I am wrong, I would like to see a local mode…

RAG: 1. First you create embeddings from your documents 2. Store that in a vector db 3. Ask what the user wants and do a search in the vector db (cosine similarity etc) 4. Feed the relevant search results to your LLM and do the usual LLM stuff with the returned embeddings and chunks of the documents

Although RAG is often implemented via vector databases to find 'relevant' content, I'm not sure that's a necessary component. I've been doing what I call RAG by finding 'relevant' content for the current prompt context via a number of different algorithms that don't use vectors.

Would you define RAG only as 'prompt optimisation that involves embeddings'?

Re: Show HN: I made an app to use local AI as daily driver

#176

Post by tveita »

loudmax wrote: post #171
tveita wrote: post #157

Earlier quoted context omitted.

RAG doesn't require much data or involve any training, it is a fancy name for "automatically paste some relevant context into the prompt" Basically if you have a database of three emails and ask when Biff wanted to meet for lunch, a RAG system would select the most relevant email based on any kind of search - embeddings are most fashionable, and create a prompt like """Given this document: , answer the question "When…

That's not how RAG works. What you're describing is something closer to prompt optimization. Sibling comment from discordance has a more accurate description of RAG. There's a longer description from Nvidia here: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...

Right, you read something nebulous about how "the LLM combines the retrieved words and its own response to the query into a final answer it presents to the user", and you think there is some magic going on, and then you click one link deeper and read at https://ai.meta.com/blog/retrieval-augmented-generation-stre... :

> Given the prompt “When did the first mammal appear on Earth?” for instance, RAG might surface documents for “Mammal,” “History of Earth,” and “Evolution of Mammals.” These supporting documents are then concatenated as context with the original input and fed to the [...] model

Finding the relevant context to put in the prompt is a search problem, nearest neighbour search on embeddings is one basic way to do it but the singular focus on "vector databases" is a bit of hype phenomenon IMO - a real world product should factor a lot more than just pure textual content into the relevancy score. Or is your personal AI assistant going to treat emails from yesterday as equally relevant as emails from a year ago?

Re: Show HN: I made an app to use local AI as daily driver

#177

Post by shanusmagnus »

wodow wrote: post #165
domano wrote: post #163

Hey, i bought it, nice work! A few things: * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Such a chat ui is not usable for me without this feature. * The feedback button does nothing for me, just changes focus to chrom…

> * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Agreed, but what I would also really like (from this and ChatGPT) would be branching: take a conversation in two different ways from some point and retain the seperate a…

1000 upvotes for you. My brain can't compute why someone hasn't made this, along with embeddings-based search that doesn't suck.

Re: Show HN: I made an app to use local AI as daily driver

#178

Post by behnamoh »

tartrate wrote: post #117

> Full Text Search. Blazingly fast search over thousands of messages. Natural language processing has come full circle and just reinvented Ctrl+F. I had to double check that a regular '90s search function was actually the thing being advertised here, and sure enough, there is a gif demonstrating exactly that.

and yet ChatGPT doesn't support it.

Re: Show HN: I made an app to use local AI as daily driver

#179

Post by karolist »

devinprater wrote: post #62

There's another one someone made for blind users like themselves and me, called Vollama (they use a mac, so VoiceOver + Llama). It's really good. I haven't tested many others for accessibility, but it has RAG and uses Ollama as backend, so works very well for me. https://github.com/chigkim/VOLlama/

Hey. I'm sorry about your condition. I feel I'm approaching blindness eventually, this is very random, but perhaps you could share any resources I could learn to prepare for this so I could continue using the web when/if it happens.

Re: Show HN: I made an app to use local AI as daily driver

#180

Post by toomuchtodo »

xyc wrote: post #73
toomuchtodo wrote: post #50

Hey! This is awesome! How hard would it be to plug it into something like Raindrop.io (bookmark manager) to train on all bookmarks collected?

haven't tried Raindrop.io, looks neat! Saw some other posts mentioning bookmarks as well. I'll keep this in thought, but will have to try it out first to find out.

Appreciate it, thank you.
Post reply on HN