Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector indexing all of Wikipedia on a laptop

foojay.io

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Vector indexing all of Wikipedia on a laptop

#111

Post by vineyardmike »

janalsncm wrote: post #53
traverseda wrote: post #12

Earlier quoted context omitted.

This is pretty early in the game to be relying on proprietary embeddings, don't you think? If if they are 20% better, blink and there will be a new normal. It's insane to me that someone, this early in the gold rush, would be mining in someone else's mine, so to speak

It’s not just that. Embeddings aren’t magic. If you’re going to be creating embeddings for similarity search, the first thing you need to ask yourself is what makes two vectors similar such that two embeddings should even be close together? There are a lot of related sources of similarity, but they’re slightly different. And I have no idea what Cohere is doing. Additionally, it’s not clear to me how queries can and s…

> If you’re going to be creating embeddings for similarity search, the first thing you need to ask yourself is what makes two vectors similar such that two embeddings should even be close together?

I have no association with Cohere, but in their docs clearly say that their embedding were trained so two similar vectors have similar "semantic meaning". Which is still pretty vague, but it's at least clear what their goals were.

> Selling “embeddings as a service” is a bit like selling hashing as a service.

Coincidentally, Cohere also aggressively advertises that they want you to fine-tune and co-develop custom models (with their proprietary services).

Re: Vector indexing all of Wikipedia on a laptop

#113

Post by nostrebored »

vineyardmike wrote: post #111
janalsncm wrote: post #53

Earlier quoted context omitted.

It’s not just that. Embeddings aren’t magic. If you’re going to be creating embeddings for similarity search, the first thing you need to ask yourself is what makes two vectors similar such that two embeddings should even be close together? There are a lot of related sources of similarity, but they’re slightly different. And I have no idea what Cohere is doing. Additionally, it’s not clear to me how queries can and s…

> If you’re going to be creating embeddings for similarity search, the first thing you need to ask yourself is what makes two vectors similar such that two embeddings should even be close together? I have no association with Cohere, but in their docs clearly say that their embedding were trained so two similar vectors have similar " semantic meaning ". Which is still pretty vague, but it's at least clear what their g…

But this is the GPs point — that doesn’t mean they’re optimized for retrieval.

Re: Vector indexing all of Wikipedia on a laptop

#114

Post by nostrebored »

rahimnathwani wrote: post #50
minimaxir wrote: post #18

Earlier quoted context omitted.

You can do much bigger chunks with models that support RoPE embeddings, such as nomic-embed-text-1.5 which has a 8192 context length: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 In theory this would be an efficiency boost but the performance math can be tricky.

Why would you want chunks that big for vector search? Wouldn't there be too much information in each chunk, making it harder to match a query to a concept within the chunk?

The problem is that often semantic meaning depends on state multiple paragraphs or sections away.

This is a coarse way to tackle that

Re: Vector indexing all of Wikipedia on a laptop

#117

Post by [deleted] »

jbellis wrote: post #103
HammadB wrote: post #89

"The obstacle is that until now, off-the-shelf vector databases could not index a dataset larger than memory, because both the full-resolution vectors and the index (edge list) needed to be kept in memory during index construction. Larger datasets could be split into segments, but this means that at query time they need to search each segment separately, then combine the results, turning an O(log N) search per segmen…

I was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S is proportional to N.

[deleted]

Re: Vector indexing all of Wikipedia on a laptop

#118

Post by solarengineer »

gfourfour wrote: post #59
abetusk wrote: post #19

Earlier quoted context omitted.

This probably deserves its own article and might be of interest to the HN community.

I will probably make a post when I launch my app! For now I’m trying to figure out how I can host the whole system for cheap because I don’t anticipate generating much revenue

I'm interested in hearing about what you will be hosting.

Would Digital Ocean or Hetzner meet your needs?

Re: Vector indexing all of Wikipedia on a laptop

#119

Post by HammadB »

jbellis wrote: post #103
HammadB wrote: post #89

"The obstacle is that until now, off-the-shelf vector databases could not index a dataset larger than memory, because both the full-resolution vectors and the index (edge list) needed to be kept in memory during index construction. Larger datasets could be split into segments, but this means that at query time they need to search each segment separately, then combine the results, turning an O(log N) search per segmen…

I was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S is proportional to N.

Sure, I see. I think this is an area where complexity analysis doesn’t lead to useful information.

To be more correct it’s O(N/C log C) where C is the capacity of a segment. In this case you can ignore 1/C and log C as constant. So now sure, you actually just have O(N). But this is not super useful as it says that a segmented hnsw approach and brute force approach are the same - when this is really not the case in practice.

Also O(N log N) > O(N) so I’m not sure why we would ever do anything with segmentation according to that analysis if it were correct.

Re: Vector indexing all of Wikipedia on a laptop

#120

Post by lionkor »

traverseda wrote: post #7

>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.

Could just turn down swappiness?!
Post reply on HN