Earlier quoted context omitted.
I've found that 512 characters, more or less, is a good length for embeddings. More than that, and a bunch of models will just ignore the text. Less than half that, there isn't always enough context in the text to be able to group the chunks by keyterm or other labels. I do this type of additive context using the first few results from the embedding search. If it's tweets, it makes sense to use shorter chunks. I was…
That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy. Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk. I imagine it how I might flip though a book using the index and also table of contents an…
One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...