Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dot – A standalone open source app meant for easy use of local LLMs and RAG

github.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#31

Post by gavmor »

Thanks for sharing! I look forward to playing with this once I get off my phone. Took a look at the code, though, to see if you've implemented any of the tricks I've been too lazy to try.

`text_splitter=RecursiveCharacterTextSplitter( chunk_size=8000, chunk_overlap=4000)`

Does this simple numeric chunking approach actually work? Or are more sophisticated splitting rules going to make a difference?

`vector_store_ppt=FAISS.from_documents(text_chunks_ppt, embeddings)`

So we're embedding all 8000 chars behind a single vector index. I wonder if certain documents perform better at this fidelity than others. To say nothing of missed "prompt expansion" opportunities.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#32

Post by spxneo »

prosunpraiser wrote: post #13
eole666 wrote: post #11

Looks nice! But some informations about the hardware requirement are often missing in this kind of project : - how much ram is needed - what CPU do you need for decent performances - can it run on a GPU? And if it does how much vram do you need / does it work only on Nvidia?

Not sure if this helps but this is from tinkering with Mistral 7B on both my M1 Pro (10 Core, 16 GB RAM) and WSL 2 w/ CUDA (Acer Predator 17, i7-7700HK, GTX 1070 Mobile, 16GB DRAM, 8GB VRAM). - Got 15 - 18 Tokens / sec on WSL 2 with slightly higher on M1. Can think of that to about 10 - 15 words per second. Both were using GPU. Haven’t tried CPU on M1 but on WSL 2 it was low single digits - super slow for anything pr…

how does 7b match up to Mistral 8x7B?

coming from chatgpt4 it was a huge breath of fresh air to not deal with the judeo-christian biased censorship.

i think this is the ideal localllama setup--uncensored, unbiased, unlimited (only by hardware) LLM+RAG

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#33

Post by simonw »

turnsout wrote: post #22
simonw wrote: post #19

Earlier quoted context omitted.

What options do you think work better?

I don’t have an opinion, just wondering why they didn’t choose other another option such as Sentence Embeddings, OpenAI embeddings, etc.

FAISS can be used with OpenAI embeddings (and any other embedding model).

FAISS is technology for fast indexed similarity vector search - using it is an independent decision from which model you use to create those vectors.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#34

Post by verdverm »

victor106 wrote: post #16
verdverm wrote: post #15

Earlier quoted context omitted.

https://www.kapa.ai/ seems to be the most popular saas for developer tools & docs. I'm seeing it all over the place

Used it, it’s just glorified marketing and among all the solutions we tried it ranked in the bottom three. The best at least for now is to just use OpenAI’s custom gpt and with some clever (but not hard) it’s quite good.

If you want to allocate resources to building out the AI, connecting and ingesting sources, setting up rag, fine tuning and hyper param optimization...

Most companies lack the expertise and resources. Kapa means they get a docs bot while maintaining focus on what they do best.

Kapa must be doing something right since they seem to be growing. Having used it in a few discords, it's what I'd expect for quality for a saas product built on current ai capabilities.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#35

Post by andeee23 »

reacharavindh wrote: post #3

I’m curious to try it out. There seem to be many options to upload a document and ask stuff about it. But, the holy grail is an LLM that can successfully work on a large corpus of documents and data like slack history, huge wiki installations and answer useful questions with proper references. I tried a few, but they don’t really hit the mark. We need the usability of a simple search engine UI with private data sourc…

https://storytell.ai seems to be doing what you’re looking for, especially the part with the linking to proper references

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#36

Post by krageon »

verdverm wrote: post #34
victor106 wrote: post #16

Earlier quoted context omitted.

Used it, it’s just glorified marketing and among all the solutions we tried it ranked in the bottom three. The best at least for now is to just use OpenAI’s custom gpt and with some clever (but not hard) it’s quite good.

If you want to allocate resources to building out the AI, connecting and ingesting sources, setting up rag, fine tuning and hyper param optimization... Most companies lack the expertise and resources. Kapa means they get a docs bot while maintaining focus on what they do best. Kapa must be doing something right since they seem to be growing. Having used it in a few discords, it's what I'd expect for quality for a saa…

> Kapa must be doing something right since they seem to be growing

It's marketing. The person you responded to said they're all marketing. Saying they "must be doing something right" because other people are also falling for it is how you get scammed.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#37

Post by verdverm »

krageon wrote: post #36
verdverm wrote: post #34

Earlier quoted context omitted.

If you want to allocate resources to building out the AI, connecting and ingesting sources, setting up rag, fine tuning and hyper param optimization... Most companies lack the expertise and resources. Kapa means they get a docs bot while maintaining focus on what they do best. Kapa must be doing something right since they seem to be growing. Having used it in a few discords, it's what I'd expect for quality for a saa…

> Kapa must be doing something right since they seem to be growing It's marketing. The person you responded to said they're all marketing. Saying they "must be doing something right" because other people are also falling for it is how you get scammed.

That's what they say. What I see is high engagement of users in discord channels.

Even OpenAI, GP's alternative, is listed as using Kapa... and no public sign ups available yet either

I saw a glimpse of the internal dashboard companies get. It's much more than just question answering. Another big piece is the feedback, seeing user interaction, and being able to improve things over time

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#38

Post by 0x008 »

gavmor wrote: post #31

Thanks for sharing! I look forward to playing with this once I get off my phone. Took a look at the code, though, to see if you've implemented any of the tricks I've been too lazy to try. `text_splitter=RecursiveCharacterTextSplitter( chunk_size=8000, chunk_overlap=4000)` Does this simple numeric chunking approach actually work? Or are more sophisticated splitting rules going to make a difference? `vector_store_ppt=F…

Of all the off the shelf text splitters I have tried, the recursive character splitter actually performs really well. Especially if the chunk size is so large you will likely have more than the actual needed context in a chunk anyway.

Regarding the index usually a mix of BM25 and vector index seems to perform best for most generic data.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#39

Post by 0x008 »

MasterYoda wrote: post #27

I have collected so much information in text files on my computer that it has become unmanageable to find anything. Now with local AI solutions, I wondered if I could create a smart search engine that could provide answers to the information that exists on my personal data. My question is. 1 - Even if there is so much data that I can no longer find stuff, how much text data is needed to train an LLM to work ok? Im no…

/r/localllama is probably the place where you want to ask your questions. They are very up to date and lots of good recommendations there.

Re: Dot – A standalone open source app meant for easy use of local LLMs and RAG

#40

Post by cyanydeez »

reacharavindh wrote: post #3

I’m curious to try it out. There seem to be many options to upload a document and ask stuff about it. But, the holy grail is an LLM that can successfully work on a large corpus of documents and data like slack history, huge wiki installations and answer useful questions with proper references. I tried a few, but they don’t really hit the mark. We need the usability of a simple search engine UI with private data sourc…

If anyone wants down this golden path, I'd recommend forking open search server. It's quite a feat and does the crawling part well.

https://www.opensearchserver.com/

Post reply on HN