We really need a better search engine for obsidian notes, the default search is horrendous
What would you like to see? (I work on Obsidian)
Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
61–70 of 72 posts
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#62Earlier quoted context omitted.
You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.
If you're concatenating after chunking , then the overlapping windows add quite a lot of repetition. Also, if it cuts off mid-json / mid-structured output then overlapping windows once again cause issues. Define a custom recursive text splitter in langchain, and do chunking heuristically. It works a lot better. That being said, it is useful to maintain some global and local context. But, I wouldn't use overlapping wi…
In place of simply concatenating after chunking, a more effective approach might be to retrieve and return the corresponding segments from the original documents that are relevant to the context. For instance, if we're dealing with short pieces of text such as Hacker News comments, it's fairly straightforward. Any partial match can prompt the return of the entire comment as it is.
When working with more extensive documents, the process gets a bit more intricate. In this case, your embedding database might need to hold more information per entry. Ideally, for each document, the database should store identifiers like the document ID, the starting token number, and the ending token number. This way, even if a document appears more than once among the top results from a query, it's possible to piece together the full relevant excerpt accurately.
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#63Earlier quoted context omitted.
Does your data stay local by doing this?
Mostly but it does upload some of the vectorized data to insert into the prompt for context. When you do a query llama-index tries to discover content related to your prompt and injects it for context so its not entirely local.
Are you calculating your embeddings locally, and using OpenAI's APIs only for text generation?
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#64Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#65> We start by parsing documents into chunks. A sensible default is to chunk documents by token length, typically 1,500 to 3,000 tokens per chunk. However, I found that this didn’t work very well. A better approach might be to chunk by paragraphs (e.g., split on \n\n). Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.
You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.
This is only a good idea if you are *specifically not* using OpenAI.
If you use local models then it's a fantastic idea.
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#66A tool for Obisidian which no has yet made: AI memory palace illustrations There would be a simple system for rooms and the AI / program would edit them and things to them which when clicked on could lead to new "places"
Can you elaborate? Rooms + exits can already be done by creating links.
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#67This looks great! I was about to start learning and diving into Obsidian about a month ago, finally driven to begin building a personal knowledgebase... And then I found Mem.ai and dove into that instead, and i've been extremely happy with it. It accomplishes this aspect he's offering here (where it uses your knowledgebase to assist in your writing). However, it's also got built in chat with your knowledgebase, and h…
I'm not sure a offline-first document editor is comparable to a hosted SaaS about AI. This plugin is one of many, while mem.ai is non-customizable tool where someone else owns your data and seems to offer no data portability.
I was excited about Mem a few years ago. But was disappointed they did not support math/MathJax notation ($…$) despite millions raised.
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#68Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#69I love obsidian but the first plugin I tried (and paid for) led to subtle data loss and resulted in many hours of checking and merging a month of backups. Not going to risk that again.
Wow, that's awful -- and an extreme outlier. Most plugins are free. And given under the hood Obsidian notes are just markdown files on the local filesystem, backups can be managed by git or TimeMachine or rsync or whatever else you might use on other directories. That's not to discredit your experience, just speaking up for the sake of others who might be unduly scared off.
Sure, this one had a free version and it was great. Except after a few weeks Obsidian started behaving weirdly, and was 100% reproducible by enabling or disabling the plugin in question. And then I noticed that some of the markdown files were modified (sections deleted) - but I hadn't noticed immediately, so they'd had edits after they were modified - hence the tedious manual merge.
So it may be "just this one plugin", but Obsidian is so important that I'm just not willing to risk it.
Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking
#70Earlier quoted context omitted.
He said he’d like to see a better search engine
I was hoping for something more descriptive than “better”
Semantic search. The current search feels like it is barely doing something smarter than substring matching / basic regex. The search should be more like Google and less like matching substrings