Vector indexing all of Wikipedia on a laptop • Hacker News Board

Re: Vector indexing all of Wikipedia on a laptop

#101

Post by Atotalnoob » Thu, May 30, 2024, 1:33 AM UTC

lemarchr wrote: post #100

Atotalnoob wrote: post #97

Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird

See https://www.datastax.com/our-people/jonathan-ellis Wikipedia lists them as a founder. Perhaps their author bio is outdated, or Wikipedia is. Not sure about your friend.

They were definitely a founder, but they are not the current cto

Re: Vector indexing all of Wikipedia on a laptop

#102

Post by Atotalnoob » Thu, May 30, 2024, 1:34 AM UTC

metadat wrote: post #99

Atotalnoob wrote: post #97

Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird

What are you talking about? The datastax site lists it: > SANTA CLARA, Calif. – September 28, 2020 – DataStax today announced that DataStax Co-Founder and CTO Jonathan Ellis will deliver a keynote address at ApacheCon @Home 2020 https://www.datastax.com/press-release/datastax-co-founder-a... . As an aside, I'm an ApacheCon presenter but there was no press release about the hot excitement of my involvement. Maybe next…

That’s from 2024. They aren’t the cto of datastax currently

Re: Vector indexing all of Wikipedia on a laptop

#103

Post by jbellis » Thu, May 30, 2024, 1:35 AM UTC

HammadB wrote: post #89

"The obstacle is that until now, off-the-shelf vector databases could not index a dataset larger than memory, because both the full-resolution vectors and the index (edge list) needed to be kept in memory during index construction. Larger datasets could be split into segments, but this means that at query time they need to search each segment separately, then combine the results, turning an O(log N) search per segmen…

I was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S is proportional to N.

Re: Vector indexing all of Wikipedia on a laptop

#104

Post by metadat » Thu, May 30, 2024, 1:37 AM UTC

Atotalnoob wrote: post #102

metadat wrote: post #99

Earlier quoted context omitted.

What are you talking about? The datastax site lists it: > SANTA CLARA, Calif. – September 28, 2020 – DataStax today announced that DataStax Co-Founder and CTO Jonathan Ellis will deliver a keynote address at ApacheCon @Home 2020 https://www.datastax.com/press-release/datastax-co-founder-a... . As an aside, I'm an ApacheCon presenter but there was no press release about the hot excitement of my involvement. Maybe next…

That’s from 2024. They aren’t the cto of datastax currently

Maybe it's the highest rank they achieved and still trying to capitalize on it. So what? Should they now say "Unemployed"?

There has to be more interesting things to discuss than this.

P.s. I think you meant 2020.

Re: Vector indexing all of Wikipedia on a laptop

#105

Post by jbellis » Thu, May 30, 2024, 1:37 AM UTC

m3kw9 wrote: post #95

He should have asked HN on the cheapest way to embed Wikipedia before starting

I'm baffled that so many people fixate on the estimated cost and miss the fact that it's a public dataset. As in, free.

Re: Vector indexing all of Wikipedia on a laptop

#106

Post by jbellis » Thu, May 30, 2024, 1:43 AM UTC

StrangeDoctor wrote: post #90

lilatree wrote: post #82

“… turning an O(log N) search per segment into O(N) overall.” Can someone explain why?

When it’s all in memory you get to amortize the cost of the initial load. Or just pay it when it’s not part of the hot path. When it’s segmented, you’re doing that because memory is full and you need to read in all the segments you don’t have. That’ll completely overwhelm the log n of the search you still get

I was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S (number of segments) is proportional to N (number of vectors).

Re: Vector indexing all of Wikipedia on a laptop

#107

Post by jbellis » Thu, May 30, 2024, 1:56 AM UTC

Atotalnoob wrote: post #97

Why is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people That’s kind of weird

I guess I'm kind of a CTO emeritus now -- I mostly write code, by choice. https://github.com/jbellis

Re: Vector indexing all of Wikipedia on a laptop

#108

Post by syllogistic » Thu, May 30, 2024, 2:05 AM UTC

jbellis wrote: post #105

m3kw9 wrote: post #95

He should have asked HN on the cheapest way to embed Wikipedia before starting

I'm baffled that so many people fixate on the estimated cost and miss the fact that it's a public dataset . As in, free.

It's in the first sentence of the article too :)

Re: Vector indexing all of Wikipedia on a laptop

#109

Post by Khelavaster » Thu, May 30, 2024, 2:33 AM UTC

This is how Microsoft powered it's academic paper search in 2016, before rolling it into Bing in 2020!

Re: Vector indexing all of Wikipedia on a laptop

#110

Post by m3kw9 » Thu, May 30, 2024, 3:08 AM UTC

jbellis wrote: post #105

m3kw9 wrote: post #95

He should have asked HN on the cheapest way to embed Wikipedia before starting

I'm baffled that so many people fixate on the estimated cost and miss the fact that it's a public dataset . As in, free.

Getting the embeddings ain’t free