91–100 of 146 posts
"The obstacle is that until now, off-the-shelf vector databases could not index a dataset larger than memory, because both the full-resolution vectors and the index (edge list) needed to be kept in memory during index construction. Larger datasets could be split into segments, but this means that at query time they need to search each segment separately, then combine the results, turning an O(log N) search per segmen…
>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
[article author] TBH this was sloppy on my part. I tested multiple runs of the index build and early on kswapd was super busy. I assumed Linux was just caching recently read parts of the source dataset, but it's also possible it was something external to the index build since it's my daily driver machine. After I turned off swap I had no issues and didn't look into it harder.
Edit: see for instance https://insights.oetiker.ch/linux/fadvise.html
Earlier quoted context omitted.
Can't mlock be wrapped out in a safe API?
Probably, but I'm not aware of any Java library that provides that functionality.
Otherwise, JNA is probably the easiest way, and how Cassandra does it.
https://docs.oracle.com/en/java/javase/21/core/calling-c-lib...
He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people
That’s kind of weird
Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird
Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird
> SANTA CLARA, Calif. – September 28, 2020 – DataStax today announced that DataStax Co-Founder and CTO Jonathan Ellis will deliver a keynote address at ApacheCon @Home 2020
https://www.datastax.com/press-release/datastax-co-founder-a....
As an aside, I'm an ApacheCon presenter but there was no press release about the hot excitement of my involvement. Maybe next time :)
Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird
Wikipedia lists them as a founder. Perhaps their author bio is outdated, or Wikipedia is. Not sure about your friend.