90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
1–10 of 39 posts
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#2Curious about the "outside of the database" index generation part. Is this index WAL-protected eventually?
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#3Curious about the "outside of the database" index generation part. Is this index WAL-protected eventually?
Yes it is WAL protected: the advantage of external indexing is that the HNSW graph is being constructed externally on multiple cores instead on a single core inside the Postgres process. But eventually the graph is being parsed and processed inside Postgres with all the necessary WAL logs for blocks.
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#4How does performance scale (vs pgvector) when you have an index and start loading data in parallel? Or how does this scale vs the to-be-released pgvector 0.5.2?
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#5As someone who just indexed 6m documents with pgvector, I can say it’s a massive time sync - on the order of days, even with a 32 core 64Gb RDS instance.
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#6How does performance scale (vs pgvector) when you have an index and start loading data in parallel? Or how does this scale vs the to-be-released pgvector 0.5.2?
I'm also concerned about these (tested!) errors:
> https://github.com/lanterndata/lantern/blob/040f24253e5a2651...
> Operator can only be used inside of an index
Isn't the use of the distance operator in scan+sort critical for generating the expected/correct result that's needed for validating the recall of an ANN-only index?
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#7So approximately 0% chance I could use this on AWS RDS or Aurora correct?
Still, very impressive
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#8You piqued my interest enough to sign up and try...but now it needs an Access Code to try the DB, any HN special here?
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#9You piqued my interest enough to sign up and try...but now it needs an Access Code to try the DB, any HN special here?
Try YCW24! :)
Re: 90x Faster Than Pgvector – Lantern's HNSW Index Creation Time
#10How does performance scale (vs pgvector) when you have an index and start loading data in parallel? Or how does this scale vs the to-be-released pgvector 0.5.2?
I'm also concerned about these (tested!) errors: > https://github.com/lanterndata/lantern/blob/040f24253e5a2651... > Operator can only be used inside of an index Isn't the use of the distance operator in scan+sort critical for generating the expected/correct result that's needed for validating the recall of an ANN-only index?
Ah, thank you for noticing! We actually have typo in the error message. It actually should be the operator instead of .
There's some context on the operator here: https://github.com/lanterndata/lantern?tab=readme-ov-file#a-...