Imagine the marketing coup, when we’re all saying “Machine learning? Eh, it’s all just a bunch of Dot’s products.”
Dot – A standalone open source app meant for easy use of local LLMs and RAG
21–30 of 43 posts
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#22Curious about the choice of FAISS. It's a bit older now, and there are many more options for creating and selecting embeddings. Does FAISS still offer some advantages?
What options do you think work better?
I don’t have an opinion, just wondering why they didn’t choose other another option such as Sentence Embeddings, OpenAI embeddings, etc.
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#23Not sure if install the Windows GPU or CPU app version [1].
I have:
Processor: Ryzen 5 3600
Video card: Geforce GTX 1660 TI 6Gb DDR6 (Zotac)
RAM: 16Gb DDR4 2666mhz
Any recommendations?
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#24Not sure if install the Windows GPU or CPU app version [1]. I have: Processor: Ryzen 5 3600 Video card: Geforce GTX 1660 TI 6Gb DDR6 (Zotac) RAM: 16Gb DDR4 2666mhz Any recommendations? [1] https://dotapp.uk/download.html
With those settings I would recommend GPU. CUDA acceleration really makes it faster, but keep in mind the CUDA toolkit 12.2 install will be a some 3-4gb
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#25Curious about the choice of FAISS. It's a bit older now, and there are many more options for creating and selecting embeddings. Does FAISS still offer some advantages?
Hi! I'm the guy who made Dot. I remember experimenting with a few different vector stores in the early stages of the project but decided to settle with FAISS. I mainly chose it because it made it easy to perform the whole embedding process locally and also because it allows to merge vector stores which is what I use to load multiple types of documents at once. But I am definately not an expert on the topic and would really appreciate suggestions on other alternatives that might work better! :)
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#26I’m curious to try it out. There seem to be many options to upload a document and ask stuff about it. But, the holy grail is an LLM that can successfully work on a large corpus of documents and data like slack history, huge wiki installations and answer useful questions with proper references. I tried a few, but they don’t really hit the mark. We need the usability of a simple search engine UI with private data sourc…
Have you tried https://markprompt.com/ ?
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#27I have collected so much information in text files on my computer that it has become unmanageable to find anything. Now with local AI solutions, I wondered if I could create a smart search engine that could provide answers to the information that exists on my personal data.
My question is.
1 - Even if there is so much data that I can no longer find stuff, how much text data is needed to train an LLM to work ok? Im not after an AI that could answer general question, only an AI that should be able to answer what I already know exist in the data.
2 - I understand that the more structured the data are, the better, but how important is it when training an LLM with structured data? Does it just figuring stuff out anyways in a good way mostly?
3 - Any recommendation where to start, how to run an LLM AI locally, train on your own data?
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#28Looks nice! But some informations about the hardware requirement are often missing in this kind of project : - how much ram is needed - what CPU do you need for decent performances - can it run on a GPU? And if it does how much vram do you need / does it work only on Nvidia?
Right now the minimum amount of RAM I would recommend is 16gb, I think it can run with less memory but that will require a few changes here and there (although they might reduce performance). I would also strongly recommend using a GPU over CPU, in my experience it can make the LLM run twice as fast if not more. Only Nvidia GPUs are supported for now and the CUDA toolkit 12.2 is required to run Dot.
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#29Not sure if install the Windows GPU or CPU app version [1]. I have: Processor: Ryzen 5 3600 Video card: Geforce GTX 1660 TI 6Gb DDR6 (Zotac) RAM: 16Gb DDR4 2666mhz Any recommendations? [1] https://dotapp.uk/download.html
With those settings I would recommend GPU. CUDA acceleration really makes it faster, but keep in mind the CUDA toolkit 12.2 install will be a some 3-4gb
Thank you!
Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG
#30I have a reasonably wast library of technical/scientific epubs/documents. Could I use this to import them and the quiz the books?
Yes! Of course because the LLM is running locally it is not as advanced as bigger models like Claude or GPT, but you can definately quiz the documents. From my experience it performs better with specific questions rather than more ambigous questions that require extensive understanding of the whole document.