161–170 of 246 posts
Wow, I did not expect at all this will end up on the front page. Thank you for all the enthusiasm, I'll try to get to more questions later today but if there's something I missed my X/twitter DM is open: https://x.com/chxy
Earlier quoted context omitted.
Retrieval augmented generation. In short you use an LLM to classify your documents (or chunks from them) up front. Then when you want to ask the LLM a question you pull the most relevant ones back to feed it as additional context.
I dont get it. To my understanding it takes huge amounts of data to build any any form of RAG. Simply because it enlarges the statistical model you later prompt. If the model is not big enough how would you expect it to answer you in a non qualifying matter ? It simply can't. So I don't really buy it and I have yet to see it work better than any rdbms search index. Tell me I am wrong, I would like to see a local mode…
1. First you create embeddings from your documents
2. Store that in a vector db
3. Ask what the user wants and do a search in the vector db (cosine similarity etc)
4. Feed the relevant search results to your LLM and do the usual LLM stuff with the returned embeddings and chunks of the documents
A few things:
* The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point.
Such a chat ui is not usable for me without this feature.
* The feedback button does nothing for me, just changes focus to chrome.
* The LLaVA model tells me that it can not generate images since it is a text based AI model. My prompts were "Generate an image of ..."
- Ability to use local and OpenAI models (ideally it has defaults for common local models)
- Chat UX
- Where I can point it to my JS/TS codebase
- It indexes the whole thing including dependencies for RAG. Ideally indexing has some form of awareness of model context length.
- I can use it for codegen / debugging.
The closest I have found has been aider, but it's python and I get into general python hell every time I try and run it.
Would appreciate a suggestion.
Hey, i bought it, nice work! A few things: * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Such a chat ui is not usable for me without this feature. * The feedback button does nothing for me, just changes focus to chrom…
Agreed, but what I would also really like (from this and ChatGPT) would be branching: take a conversation in two different ways from some point and retain the seperate and shared history.
I'm not sure what the UI should be. Threads? (like mail or Usenet)
Earlier quoted context omitted.
You can see ms/token in a tiny font on the top of the screen, once the text generation completes in both the videos I'd linked to. Performance will vary by machine. On my 64GB M2 Mac Studio Max, I get ~47 tokens/s (21.06ms/token) with Mistral Instruct v0.2 and ~33 tokens/s (30.14ms/token) with Mixtral Instruct v0.1.
Interesting! What's the prompt eval processing speed like compared to llama.cpp and kin?
Hey, i bought it, nice work! A few things: * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Such a chat ui is not usable for me without this feature. * The feedback button does nothing for me, just changes focus to chrom…
Because it can't generate images, it can only describe images provided by the user.
> Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it. The app looks great! Likewise, if you have any requests or ideas for improving llama.cpp, please don't hesitate to open an issue / discussion in the repo
Hey, i bought it, nice work! A few things: * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Such a chat ui is not usable for me without this feature. * The feedback button does nothing for me, just changes focus to chrom…
> * The main thing that makes ChatGPTs ui useful to me is the ability to change any of my prompts in the conversation & it will then go back to that part of the converation and regenerate, while removing the rest of the conversation after that point. Agreed, but what I would also really like (from this and ChatGPT) would be branching: take a conversation in two different ways from some point and retain the seperate a…
Out of curiosity – how is this app built?:-) There is a demo clip with a vertical scroll bar which does not fade out as it would do in a native mac app:)
Yeah I am curious what the app is built with. I saw someone mention it's using Electron, so that's a start.