Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

eBay’s Fast Billion-Scale Vector Similarity Engine

tech.ebayinc.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#11

Post by Duplo »

almog wrote: post #5

One of the best things about eBay compared to other market places like Amazon and Aliexpress, is that their search operators include not just AND (space) but also OR (comma separated values inside parentheses) and NOT (minus) but more importantly - eBay's search engine respects the supported operators and denoting which items are search results and which are suggested/similar. This cannot be said about Amazon, Aliexp…

I dislike eBay search, for example all of a sudden one of my daily bookmarked searches went from the usual 10 results to THOUSANDS, all of them not including my search term, this is the link: https://www.ebay.co.uk/sch/i.html?_nkw=Tasmota&LH_PrefLoc=1&...

Why are some sellers having all their listings (car parts) included in my search for "Tasmota"?

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#12

Post by MontyCarloHall »

cjbgkagh wrote: post #4
Scoundreller wrote: post #2

Nice to see eBay invest further on its search side. This paper is too dense for me to read through at this moment, but eBay must be a real disappointment for its shareholders: Despite its head start at being a platform for online retail open to anyone, eBay stock has only doubled in 10 years, or in other words, added $12b in equity value (currently $24b). Imagine that: a successful tech company with a real product in…

Ebay used to have an engineering limit on the number of items in their indexes so they couldn’t grow and had to increase fees instead to keep the item count down, I don’t know if they still have that limit, I haven’t been paying attention. It meant they ended up with a focus on very high margin crap, Amazon is now going the same way because they can and it makes a lot of money. Maybe with more listings eBay can lower…

>Ebay used to have an engineering limit on the number of items in their indexes so they couldn’t grow and had to increase fees instead to keep the item count down

I find this incredibly hard to believe. The hard limit on the number of total listings was deemed such an intractable problem to solve that rather than focus all engineering efforts for a few months on raising it, they just threw up their hands, didn’t even attempt to solve it, and tried to increase revenue solely through higher listing fees?

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#13

Post by bliteben »

MontyCarloHall wrote: post #12
cjbgkagh wrote: post #4

Earlier quoted context omitted.

Ebay used to have an engineering limit on the number of items in their indexes so they couldn’t grow and had to increase fees instead to keep the item count down, I don’t know if they still have that limit, I haven’t been paying attention. It meant they ended up with a focus on very high margin crap, Amazon is now going the same way because they can and it makes a lot of money. Maybe with more listings eBay can lower…

>Ebay used to have an engineering limit on the number of items in their indexes so they couldn’t grow and had to increase fees instead to keep the item count down I find this incredibly hard to believe. The hard limit on the number of total listings was deemed such an intractable problem to solve that rather than focus all engineering efforts for a few months on raising it, they just threw up their hands, didn’t even…

I find it very credible. Early ebay had very high fees on photos. They also would purge old auctions within at least 90 days. They still do purge old auctions which clearly would have some seo value. Also the whole way ebay motors was operated as essentially a separate site lends credibility as well.

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#14

Post by almog »

Duplo wrote: post #11
almog wrote: post #5

One of the best things about eBay compared to other market places like Amazon and Aliexpress, is that their search operators include not just AND (space) but also OR (comma separated values inside parentheses) and NOT (minus) but more importantly - eBay's search engine respects the supported operators and denoting which items are search results and which are suggested/similar. This cannot be said about Amazon, Aliexp…

I dislike eBay search, for example all of a sudden one of my daily bookmarked searches went from the usual 10 results to THOUSANDS, all of them not including my search term, this is the link: https://www.ebay.co.uk/sch/i.html?_nkw=Tasmota&LH_PrefLoc=1&... Why are some sellers having all their listings (car parts) included in my search for "Tasmota"?

That's interesting, I've seen it happening randomly that a search query would return unrelated results for 1 out of 100 queries using the same query.

In your example however, it seems to me that if you add quotes, the irrelevant results are gone: https://www.ebay.co.uk/sch/i.html?_nkw=%22Tasmota%22

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#15

Post by MontyCarloHall »

bliteben wrote: post #13
MontyCarloHall wrote: post #12

Earlier quoted context omitted.

>Ebay used to have an engineering limit on the number of items in their indexes so they couldn’t grow and had to increase fees instead to keep the item count down I find this incredibly hard to believe. The hard limit on the number of total listings was deemed such an intractable problem to solve that rather than focus all engineering efforts for a few months on raising it, they just threw up their hands, didn’t even…

I find it very credible. Early ebay had very high fees on photos. They also would purge old auctions within at least 90 days. They still do purge old auctions which clearly would have some seo value. Also the whole way ebay motors was operated as essentially a separate site lends credibility as well.

That just means they were trying to cut costs (by reducing their storage footprint) and increase revenue (by charging a premium for photos) at the same time. I guess the brass at eBay deemed the commercial value of archiving old listings to be worth less than the cost of storing them. Remember, storage used to be very expensive, and no amount of engineering effort could decrease the cost of server racks and hard drives.

eBay Motors is separate because the process of buying a car is very different from buying knickknacks. The number of listings on eBay Motors is tiny compared to the main site, so it doesn’t even make sense that they’d split it off specifically to split up their listing database.

None of what you mention is evidence of a technical limitation.

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#16

Post by binarymax »

Good to see another engine leverage ScANN outside of Google.

HNSW uses lots of RAM and it’s interesting how all the major engines settled on that algo.

I’m interested in how they apply filtering, since with codebook based similarity such as PQ and ScANN it’s not trivial.

Maybe one day we’ll also see someone implement a production ready Vamana engine too, which also does really well at the billion scale.

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#18

Post by cjbgkagh »

MontyCarloHall wrote: post #15
bliteben wrote: post #13

Earlier quoted context omitted.

I find it very credible. Early ebay had very high fees on photos. They also would purge old auctions within at least 90 days. They still do purge old auctions which clearly would have some seo value. Also the whole way ebay motors was operated as essentially a separate site lends credibility as well.

That just means they were trying to cut costs (by reducing their storage footprint) and increase revenue (by charging a premium for photos) at the same time. I guess the brass at eBay deemed the commercial value of archiving old listings to be worth less than the cost of storing them. Remember, storage used to be very expensive, and no amount of engineering effort could decrease the cost of server racks and hard driv…

And making a website that doesn’t suck shouldn’t be hard either… the eBay corporate culture is really bad, most blame the Meg Whitman era for that. Sometimes companies get so bad that even very simple things become insurmountable. Auction items have a few extra dimensions of complexity over normal search as the value changes quite dramatically over time and mostly at the last minute so it’s quite time sensitive. I know people who work there, they’ve checked out and are just collecting a paycheck and I don’t blame them.

Re: eBay’s Fast Billion-Scale Vector Similarity Engine

#20

Post by esafak »

jakearmitage wrote: post #17
esafak wrote: post #3

HNSW seems better in the criteria that matter. I think similarity search is a commodity now; I would not invest in developing an in-house solution given the abundance of good commercial solutions.

Such as...?

Qdrant, milvus, weaviate
Post reply on HN