61–70 of 146 posts
Maybe I’m missing something but I’ve created vector embeddings for all of English Wikipedia about a dozen times and it costs maybe $10 of compute on Colab, not $5000
Also, if you’re spending $5000 to compute embeddings, why are you indexing them on a laptop?
Earlier quoted context omitted.
Nothing too crazy, just downloading a dump, splitting it into manageable batch sizes, and using a lightweight embedding model to vectorize each article. Using the best GPU available on colab it takes maybe 8 hours if I remember correctly? Vectors can be saved as NPY files and loaded into something like FAISS for fast querying.
Did you chunk the articles? If so, in what way?
Earlier quoted context omitted.
Ah didn’t realize it was every language. Yes I’m using a light weight open model - but also my use case doesn’t require anything super heavy weight. Wikipedia articles are very feature-dense and differentiable from one another. It doesn’t require a massive feature vector to create meaningful embeddings.
it's 35M 1024 vectors Plus the text
$5000?! I indexed all of HN for ... $50 I think. And that's tens of millions of posts.
You can demo this here: https://jvectordemo.com:8443/ GH Project: https://github.com/jbellis/jvector
Earlier quoted context omitted.
Did you chunk the articles? If so, in what way?
Yes. I split the text into sentence and append sentences to a chunk until the max context window is reached. The context window size is dynamic for each article so that each chunk is roughly the same size. Then I just do a mean pool of the chunks for each article.
Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia.
>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
> This is an indication to me that something has gone very wrong in your code base. I'm not sure on what planet all of these people here live that they have success with Linux swap. It's been broken for me forever and the first thing I do is disable it everywhere.
echo y >/sys/kernel/mm/lru_gen/enabled