71–80 of 146 posts
Earlier quoted context omitted.
I don't use Java often, but when I do, I have to look up again how the heap settings work.
It’s just -xXxXxX360NoScopexXxXxHeapSize=69G
(IIRC these options are used like -XX:foo=bar)
(Edit - no, but some do use : instead of = I guess)
Maybe I’m missing something but I’ve created vector embeddings for all of English Wikipedia about a dozen times and it costs maybe $10 of compute on Colab, not $5000
You can demo this here: https://jvectordemo.com:8443/ GH Project: https://github.com/jbellis/jvector
>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
Memory managed languages won’t do long term garbage collection till memory is nearly full. I suspect they just need to pass in -xmx options to the jvm to avoid this.
If you're referring to full GC you can configure how often that happens and by default it doesn't just wait until memory is nearly full.
Earlier quoted context omitted.
Memory managed languages won’t do long term garbage collection till memory is nearly full. I suspect they just need to pass in -xmx options to the jvm to avoid this.
I actually wish this were true because of more predictable gc pauses
Maybe I’m missing something but I’ve created vector embeddings for all of English Wikipedia about a dozen times and it costs maybe $10 of compute on Colab, not $5000
This is covering 300+ languages, not just English, and it's specifically using Cohere's Embed v3 embeddings, which are provided as a service and currently priced at US$0.10 per million tokens [1]. I assume if you're running on Colab you're using an open model, and possibly a relatively lighter weight one as well? [1]: https://cohere.com/pricing
As a first step they are using PQ anyways. It seems natural to just assume all English docs have the same centroid and search that subspace with hnswlib.
Earlier quoted context omitted.
This is covering 300+ languages, not just English, and it's specifically using Cohere's Embed v3 embeddings, which are provided as a service and currently priced at US$0.10 per million tokens [1]. I assume if you're running on Colab you're using an open model, and possibly a relatively lighter weight one as well? [1]: https://cohere.com/pricing
I also don’t quite understand the value of embedding all languages into the same database. If I search for “dog” do I really need to see the same article 300 times? As a first step they are using PQ anyways. It seems natural to just assume all English docs have the same centroid and search that subspace with hnswlib.