Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

eugeneyan.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#51

Post by busseio »

SmooL wrote: post #47
gwern wrote: post #45

Earlier quoted context omitted.

You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.

I've thought about doing this as well, but I haven't tried it yet. Are there any resources/blogs/information on various strategies on how to best chunk & embed arbitrary text?

I’ve been experimenting with sliding window chunking using SRT files. They’re the subtitle format for television and have 1 to _n_ sequence numbers for each chunk, along with time stamps for when the chunk should appear on the screen. Traditionally it’s two lines of text per chunk but you can make chunks of other line counts and sizes. Much of my work with this has been with SRT files that are transcriptions exported from Otter.ai; GPT-3.5 & 4 natively understand the SRT format and the concepts of the sequence numbers and time stamps, so you can refer to them or ask for confirmation of them in a prompt.

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#52

Post by screye »

gwern wrote: post #45
Kelamir wrote: post #41

> We start by parsing documents into chunks. A sensible default is to chunk documents by token length, typically 1,500 to 3,000 tokens per chunk. However, I found that this didn’t work very well. A better approach might be to chunk by paragraphs (e.g., split on \n\n). Hmm good insight there. I've done some experimenting formerly by chunk length and it's been pretty troublesome due to missing context.

You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.

If you're concatenating after chunking , then the overlapping windows add quite a lot of repetition. Also, if it cuts off mid-json / mid-structured output then overlapping windows once again cause issues.

Define a custom recursive text splitter in langchain, and do chunking heuristically. It works a lot better.

That being said, it is useful to maintain some global and local context. But, I wouldn't use overlapping windows.

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#53

Post by SparkyMcUnicorn »

atentaten wrote: post #18
ukuina wrote: post #14

Earlier quoted context omitted.

You can use https://localai.io if you have a GPU or Apple Silicon CPU to serve up local models with an OpenAI-compatible API.

Do you know of a list of hardware recommendations to run this?

That's a loaded question, because there's different approaches you can take to run these models. Basically, you want lots of memory (ram or vram), and the more you have, the larger the models can be that you run.

I'd recommend shooting for at least 13B models.

Use "oobabooga/text-generation-webui", which can also serve an OpenAI-compatible API as well as provide a chat interface. It can serve most models, using most methods.

Check out their system requirements page[0], and join some of the communities to learn more about what hardware will work best for you.

This person[1] is providing models of all sorts, in pretty much every optimized format. They also post the minimum RAM requirements for each of the GGML models, which are best if you want to host using CPU/RAM (no video card).

[0] https://github.com/oobabooga/text-generation-webui/blob/main...

[1] https://huggingface.co/TheBloke

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#54

Post by capableweb »

shawnc wrote: post #36

This looks great! I was about to start learning and diving into Obsidian about a month ago, finally driven to begin building a personal knowledgebase... And then I found Mem.ai and dove into that instead, and i've been extremely happy with it. It accomplishes this aspect he's offering here (where it uses your knowledgebase to assist in your writing). However, it's also got built in chat with your knowledgebase, and h…

I'm not sure a offline-first document editor is comparable to a hosted SaaS about AI. This plugin is one of many, while mem.ai is non-customizable tool where someone else owns your data and seems to offer no data portability.

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#56

Post by throwaway675309 »

majormajor wrote: post #31
LouisvilleGeek wrote: post #29

Earlier quoted context omitted.

Mostly but it does upload some of the vectorized data to insert into the prompt for context. When you do a query llama-index tries to discover content related to your prompt and injects it for context so its not entirely local.

> Mostly but it does upload some of the vectorized data to insert into the prompt for context. When you do a query llama-index tries to discover content related to your prompt and injects it for context so its not entirely local. When you say "upload some of the vectorized data" do you mean in a numerical embedding form or that it will embed the original text from original similar-seeming notes directly into the prom…

The numerical vector representing the embedding is only useful for finding documents that are similar to your search query.

Those documents are then injected into your prompt and sent to some kind of LLM completion system such as GPT.

So yes, you will be sending chunks of your actual notes over the wire.

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#59

Post by gwern »

screye wrote: post #52
gwern wrote: post #45

Earlier quoted context omitted.

You don't do a sliding window? That seems like the logical way to maintain context but allow look up by 'chunks'. Embed it, say, 3 paragraphs at a time, advancing 1 paragraph per embedding.

If you're concatenating after chunking , then the overlapping windows add quite a lot of repetition. Also, if it cuts off mid-json / mid-structured output then overlapping windows once again cause issues. Define a custom recursive text splitter in langchain, and do chunking heuristically. It works a lot better. That being said, it is useful to maintain some global and local context. But, I wouldn't use overlapping wi…

I don't think the repetition is a problem. He's using a local model for human-assisted writing with pre-generated embeddings - he can use essentially an arbitrary number of embedding calls, as long as it's more useful for the human. So it's just a question of whether that improves the quality or not. (Not that the cost would be more than a rounding error to embed your typical personal wiki with something like the OA API, especially since they just dropped the prices of embeddings again.)

Re: Obsidian-Copilot: A Prototype Assistant for Writing and Thinking

#60

Post by 111111IIIIIII »

thethunderbox wrote: post #57
richard___ wrote: post #55

We really need a better search engine for obsidian notes, the default search is horrendous

In the past I have used Omnisearch which I have found to be an improvement. https://github.com/scambier/obsidian-omnisearch

This is why I don't use Obsidian. Without giving root to an aggregate of random git repos, the most important features are very subpar. I have no idea what I would even do with a fancy star graph or whatever it is, and VS Code has much better search.

Edit: Also, why is the search bar for searching all notes so buried, requiring so much effort to open? Is that because it works so poorly?

Post reply on HN