$5000?! I indexed all of HN for ... $50 I think. And that's tens of millions of posts.
Vector indexing all of Wikipedia on a laptop
31–40 of 146 posts
Re: Vector indexing all of Wikipedia on a laptop
#32>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
[article author]
TBH this was sloppy on my part. I tested multiple runs of the index build and early on kswapd was super busy. I assumed Linux was just caching recently read parts of the source dataset, but it's also possible it was something external to the index build since it's my daily driver machine. After I turned off swap I had no issues and didn't look into it harder.
Re: Vector indexing all of Wikipedia on a laptop
#33>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
As a workaround, you can mlock your process which should prevent the application pages from being evicted by swap.
FWIW this is what Cassandra does on startup, but it didn't seem worth it to go to the trouble of dealing with Unsafe for a demo project.
Re: Vector indexing all of Wikipedia on a laptop
#34>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
Yea this was strange, I've only ever seen swaps when my main memory was near being full. Maybe they're storing all embeddings in memory?
I routinely see Linux page memory out to swap while having 10+GB free. I can only guess that it really really really likes to cache recently used data from disk.
Re: Vector indexing all of Wikipedia on a laptop
#35>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
> This is an indication to me that something has gone very wrong in your code base. I'm not sure on what planet all of these people here live that they have success with Linux swap. It's been broken for me forever and the first thing I do is disable it everywhere.
On a planet where the heap size of the JVM is properly set.
Re: Vector indexing all of Wikipedia on a laptop
#36How many dimensions are in the original vectors? Something in the millions?
1024 per vector x 41M vectors
1024-dim vectors would fit into pgvector in Postgres, which can do cosine similarity indexing and doesn't require everything to fit into memory. Wonder how the performance of that would compare to this.
Re: Vector indexing all of Wikipedia on a laptop
#37Earlier quoted context omitted.
Yea this was strange, I've only ever seen swaps when my main memory was near being full. Maybe they're storing all embeddings in memory?
I routinely see Linux page memory out to swap while having 10+GB free. I can only guess that it really really really likes to cache recently used data from disk.
I've seen the same. It will sometimes prefer to swap rather than evict the disk cache.
I don't know how Linux does this in particular, but intuitively swapping can make sense if part of your allocated RAM isn't being accessed often and the disk is. The kernel isn't going to know for sure of course, and seems in my case it guessed wrong.
Re: Vector indexing all of Wikipedia on a laptop
#38Earlier quoted context omitted.
As a workaround, you can mlock your process which should prevent the application pages from being evicted by swap.
FWIW this is what Cassandra does on startup, but it didn't seem worth it to go to the trouble of dealing with Unsafe for a demo project.
Can't mlock be wrapped out in a safe API?
Re: Vector indexing all of Wikipedia on a laptop
#39> JVector, the library that powers DataStax Astra vector search, now supports indexing larger-than-memory datasets by performing construction-related searches with compressed vectors. This means that the edge lists need to fit in memory, but the uncompressed vectors do not, which gives us enough headroom to index Wikipedia-en on a laptop.
It's interesting to note that JVector accomplishes this differently than how DiskANN described doing it. My understanding (based on the links below, but I didn't read the full diff in #244) is that JVector will incrementally compress the vectors it is using to construct the index; whereas DiskANN described partitioning the vectors into subsets small enough that indexes can be built in-memory using uncompressed vectors, building those indexes independently, and then merging the results into one larger index.
OP, have you done any quality comparisons between an index built with JVector using the PQ approach (small RAM machine) vs. an index built with JVector using the raw vectors during construction (big RAM machine)? I'd be curious to understand what this technique's impact is on the final search results.
I'd also be interested to know if any other vector stores support building indexes in limited memory using the partition-then-merge approach described by DiskANN.
Finally, it's been a while since I looked at this stuff, so if I mis-wrote or mis-understood please correct me!
- DiskANN: https://dl.acm.org/doi/10.5555/3454287.3455520
- Anisotropic Vector Quantization (PQ Compression): https://arxiv.org/abs/1908.10396
- JVector/#168: How to support building larger-than-memory indexes https://github.com/jbellis/jvector/issues/168
- JVector/#244: Build indexes using compressed vectors https://github.com/jbellis/jvector/pull/244
Re: Vector indexing all of Wikipedia on a laptop
#40>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
Memory managed languages won’t do long term garbage collection till memory is nearly full.
I suspect they just need to pass in -xmx options to the jvm to avoid this.