41–50 of 72 posts
Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.
I use vim with the Copilot plugin. It's pretty astounding what it spits out. I was writing a handbook for a credit union board of directors and it was quite helpful at times.
I thought copilot only works for code. How did you use it to write a handbook?
> We start by parsing documents into chunks. A sensible default is to chunk documents by token length, typically 1,500 to 3,000 tokens per chunk. However, I found that this didn’t work very well. A better approach might be to chunk by paragraphs (e.g., split on \n\n). Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.
https://unstructured-io.github.io/unstructured/bricks.html#p...
Another option is to use llama-index and index the Obsidian vault. I use a gradio based web interface to query my Obsidian vault via GPT-3.5. It's pretty awesome.
Do you have code for this (or pointers to what you used to set this up)? I'd love to set up something similar!
I turned that into cli tool, but it isn't ready to release. Mine works like this:
qwoo index .
qwoo qa "What have I been up to?"> We start by parsing documents into chunks. A sensible default is to chunk documents by token length, typically 1,500 to 3,000 tokens per chunk. However, I found that this didn’t work very well. A better approach might be to chunk by paragraphs (e.g., split on \n\n). Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.
uses GPT 3.5, requires an OpenAI API KEY
It has prompt templates.
For bulk processing of files I still use my own python scripts.
> We start by parsing documents into chunks. A sensible default is to chunk documents by token length, typically 1,500 to 3,000 tokens per chunk. However, I found that this didn’t work very well. A better approach might be to chunk by paragraphs (e.g., split on \n\n). Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.
You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.
I use vim with the Copilot plugin. It's pretty astounding what it spits out. I was writing a handbook for a credit union board of directors and it was quite helpful at times.
This looks great! I was about to start learning and diving into Obsidian about a month ago, finally driven to begin building a personal knowledgebase... And then I found Mem.ai and dove into that instead, and i've been extremely happy with it. It accomplishes this aspect he's offering here (where it uses your knowledgebase to assist in your writing). However, it's also got built in chat with your knowledgebase, and h…
How does notion compare?
Mem.ai has integrated many aspects into it - I love that I am now unconcerned about tags or folders or categories.
I use vim with the Copilot plugin. It's pretty astounding what it spits out. I was writing a handbook for a credit union board of directors and it was quite helpful at times.
I thought copilot only works for code. How did you use it to write a handbook?