Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From grep to SPLADE: a journey through semantic search

blog.elicit.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: From grep to SPLADE: a journey through semantic search

#11

Post by salamo »

mediaman wrote: post #5

They don't mention BM25, which still outperforms much of semantic search. A fun exercise is to watch the benchmarks of the latest semantic embeddings models and see that they still struggle to match good 'ol BM25. BM25 uses the relative statistical frequency of words to identify relevant material, along with some adjustments. It doesn't use ML at all, but it works very well, especially for technical content. SPLADE i…

I've had pretty good success with BM25 + stemming, or even easier, BM25 with trigram tokenization. If the index isn't too big, the whole search can be done client-side and is lightning fast.

Re: From grep to SPLADE: a journey through semantic search

#12

Post by mozman »

delichon wrote: post #3

This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I wan…

i’m not sure if i like contextual search. a profile of me is ripe for exploitation and sale

Re: From grep to SPLADE: a journey through semantic search

#13

Post by mozman »

VivaLaPanda wrote: post #7
mediaman wrote: post #5

They don't mention BM25, which still outperforms much of semantic search. A fun exercise is to watch the benchmarks of the latest semantic embeddings models and see that they still struggle to match good 'ol BM25. BM25 uses the relative statistical frequency of words to identify relevant material, along with some adjustments. It doesn't use ML at all, but it works very well, especially for technical content. SPLADE i…

It's worse than Splade https://i.imgur.com/oGliEIg.png We've tested various hybrid approaches as well, but that's too much to go into in once post.

what’s that output from and how can i understand it? i’m happy to rtfm - just want a pointer. thanks!

Re: From grep to SPLADE: a journey through semantic search

#14

Post by throwaway81523 »

mediaman wrote: post #5

They don't mention BM25, which still outperforms much of semantic search. A fun exercise is to watch the benchmarks of the latest semantic embeddings models and see that they still struggle to match good 'ol BM25. BM25 uses the relative statistical frequency of words to identify relevant material, along with some adjustments. It doesn't use ML at all, but it works very well, especially for technical content. SPLADE i…

I found BM25 and everything resembling it (like TF/IDF) to be near useless. It was (back in the day) really necessary to use external semantic info, or at least data gathered by examining the whole document set for stuff going beyond term frequency. I was excited by the first part of the SPLADE article because I thought it was going to use LLM's to somehow find concept embeddings in documents and let you search for those. But as someone said, it turns out to be a version of synonym search except the thesaurus is generated automatically. I remember someone did that with Word2Vec some years back and it was sort of useful, but generally the problem with search systems is too many results rather than missing some that are relevant.

Re: From grep to SPLADE: a journey through semantic search

#16

Post by JadeNB »

delichon wrote: post #3

This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I wan…

> This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I want it to read the product copy and figure it out, and just tell me "We don't have any of those. But here are some others you might like..."

Why would a company want to provide this? Modern-day Amazon doesn't win when you find out that what you want isn't there; they "win" when their search is so bad that you spend a long time browsing counterfeit or not-quite-there results that you might buy. The future of an Amazon-designed chatbot that I see would be for it to try actively to snowball me into buying an inappropriate product, not to help me quickly discover that what I want isn't there.

Re: From grep to SPLADE: a journey through semantic search

#17

Post by delichon »

JadeNB wrote: post #16
delichon wrote: post #3

This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I wan…

> This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I w…

Have you seen Miracle on 34th Street, where Kris Kringle convinces Macy's and Gimbels to tell their customers that their competition has what they're looking for? That isn't as fictional as Santa Claus. I was in a Walmart yesterday when I heard a clerk tell a customer that the hardware store down the street has the thing that Walmart didn't. It happens a bazillion times a day.

Re: From grep to SPLADE: a journey through semantic search

#18

Post by mhuffman »

mediaman wrote: post #5

They don't mention BM25, which still outperforms much of semantic search. A fun exercise is to watch the benchmarks of the latest semantic embeddings models and see that they still struggle to match good 'ol BM25. BM25 uses the relative statistical frequency of words to identify relevant material, along with some adjustments. It doesn't use ML at all, but it works very well, especially for technical content. SPLADE i…

Isn't the big problem that BM25 (and friends) will help you find (and rank) exact search terms (or stemmed varieties of that search term), whereas semantic search can typically find items out-of-dictionary but "close" semantically? SPLADE, on my reading of it, seems to do a "pre-materialization" of the out-of-dictionary part.

Re: From grep to SPLADE: a journey through semantic search

#19

Post by JadeNB »

delichon wrote: post #17
JadeNB wrote: post #16

Earlier quoted context omitted.

> This feels like a big missing piece at this stage of AIs evolution. I just searched on Amazon for five inch chair casters. They used to have them but don't anymore. But that took me a long time to find out. Instead it just dumped all of the chair casters and let me read the details to find out the hard way that none of them were what I wanted across 10 pages of results. But I've been spoiled by modern chatbots. I w…

Have you seen Miracle on 34th Street, where Kris Kringle convinces Macy's and Gimbels to tell their customers that their competition has what they're looking for? That isn't as fictional as Santa Claus. I was in a Walmart yesterday when I heard a clerk tell a customer that the hardware store down the street has the thing that Walmart didn't. It happens a bazillion times a day.

> Have you seen Miracle on 34th Street, where Kris Kringle convinces Macy's and Gimbels to tell their customers that their competition has what they're looking for? That isn't as fictional as Santa Claus. I was in a Walmart yesterday when I heard a clerk tell a customer that the hardware store down the street has the thing that Walmart didn't. It happens a bazillion times a day.

One of my favorite movies, yes! And I absolutely believe that a clerk will do this; if there was ever a loyalty of staff to the retail companies for which they worked, it was probably misplaced then, and is definitely misplaced now. But that's a matter of human discretion that overseers haven't yet been able to engineer out, whereas here we're talking about something that would have specifically to engineer in. The unwillingness of Amazon even to try to confront the massive review-gaming and counterfeit-items problem leads me to believe that, to the extent that they ever viewed their mission as getting the customer what they wanted, they don't now—and so will not change their search in a way that would facilitate that.

Re: From grep to SPLADE: a journey through semantic search

#20

Post by gregw134 »

mozman wrote: post #13
VivaLaPanda wrote: post #7

Earlier quoted context omitted.

It's worse than Splade https://i.imgur.com/oGliEIg.png We've tested various hybrid approaches as well, but that's too much to go into in once post.

what’s that output from and how can i understand it? i’m happy to rtfm - just want a pointer. thanks!

It's various measures of recall rate. Recall@500 means what percentage of the time does the target document show up in the top 500 results from the retrieval system.
Post reply on HN