Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xapiand: A fast, simple, modern search and storage engine

kronuz.io

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Xapiand: A fast, simple, modern search and storage engine

#11

Post by michelpp »

Xapian has a long history starting in the early 80s:

https://xapian.org/history

I've used Xapian extensively, but not this new Xapiand tool, so I can only speak to the actual library. Xapian is a C++ library that accesses index data files directly on disk. There are bindings for various languages, say Python, let's you do 'import xapian' and get FFI bindings to the library, then you basically open your on disk index files and issue queries.

Xapian supports many concurrent readers, but only one writer. It's not a server, there are no protocols. Maybe that's what this Xapiand tool adds. In general the overhead is very, very light, just enough ram to hold the library code, the OS takes care of all the filesystem level caching.

Many of the very same concepts that are in Lucene, Documents, Terms, weights, flavors of BM25 relevance ranking, query parsing trees, relevancy operators, etc, all apply to Xapian as well.

Re: Xapiand: A fast, simple, modern search and storage engine

#12

Post by _wmd »

michelpp wrote: post #11

Xapian has a long history starting in the early 80s: https://xapian.org/history I've used Xapian extensively, but not this new Xapiand tool, so I can only speak to the actual library. Xapian is a C++ library that accesses index data files directly on disk. There are bindings for various languages, say Python, let's you do 'import xapian' and get FFI bindings to the library, then you basically open your on disk index…

I love Xapian, the quality of its recall is excellent and indexing performance very hard to find fault with. There's just a tiny problem - it's stuck with the GPL, despite a long effort to relicence the code going back years.

Re: Xapiand: A fast, simple, modern search and storage engine

#13

Post by mdnormy »

cetra3 wrote: post #7
hardwaresofton wrote: post #6

Earlier quoted context omitted.

Not sure if you know about tantivy but it's cool too: https://github.com/tantivy-search/tantivy

Also worth mentioning is Toshi: https://github.com/toshi-search/Toshi Toshi is to ElasticSearch as Tantivy is to Lucene if that makes sense. Obviously as they are new they are not at feature parity, but Tantivy does win at some benchmarks: https://tantivy-search.github.io/bench/

Non-native English speaker. But isn't easier to understand like so,

"Tantivity to Toshi, is as Lucene to Elasticsearch"

Re: Xapiand: A fast, simple, modern search and storage engine

#14

Post by Kronuz »

_wmd wrote: post #12
michelpp wrote: post #11

Xapian has a long history starting in the early 80s: https://xapian.org/history I've used Xapian extensively, but not this new Xapiand tool, so I can only speak to the actual library. Xapian is a C++ library that accesses index data files directly on disk. There are bindings for various languages, say Python, let's you do 'import xapian' and get FFI bindings to the library, then you basically open your on disk index…

I love Xapian, the quality of its recall is excellent and indexing performance very hard to find fault with. There's just a tiny problem - it's stuck with the GPL, despite a long effort to relicence the code going back years.

Maybe xapian library just needs a little push from a larger community to make relicensing faster, Xapiand could help towards that end by helping brining in more people which can help. Xapiand source code is itself licensed as MIT (before compiling), and xapian community is already taking big steps towards relicensing.

Re: Xapiand: A fast, simple, modern search and storage engine

#15

Post by bheesham »

francislavoie wrote: post #10
catmanjan wrote: post #8

I'm interested, can anyone give a quick overview of why you'd use this over Elasticsearch?

Well for one, Java is ridiculously memory hungry. The resource costs of Elasticsearch is the #1 reason I'm not using it. I've seen a few projects which had the aim of reimplementing the Elasticsearch backend in Rust, but were incomplete. That would be my ideal solution, personally.

I don't know how true that is anymore, that "Java is ridiculously memory hungry".

It does power billions of devices, after all. :-P

Re: Xapiand: A fast, simple, modern search and storage engine

#16

Post by gshack »

bheesham wrote: post #15
francislavoie wrote: post #10

Earlier quoted context omitted.

Well for one, Java is ridiculously memory hungry. The resource costs of Elasticsearch is the #1 reason I'm not using it. I've seen a few projects which had the aim of reimplementing the Elasticsearch backend in Rust, but were incomplete. That would be my ideal solution, personally.

I don't know how true that is anymore, that "Java is ridiculously memory hungry". It does power billions of devices, after all. :-P

So does the diesel engine, but's not eco friendly.

Re: Xapiand: A fast, simple, modern search and storage engine

#17

Post by ousta »

bheesham wrote: post #15
francislavoie wrote: post #10

Earlier quoted context omitted.

Well for one, Java is ridiculously memory hungry. The resource costs of Elasticsearch is the #1 reason I'm not using it. I've seen a few projects which had the aim of reimplementing the Elasticsearch backend in Rust, but were incomplete. That would be my ideal solution, personally.

I don't know how true that is anymore, that "Java is ridiculously memory hungry". It does power billions of devices, after all. :-P

why do you think they called their product "elastic"? java heap size? 32gb? above that and you are in for problems

Re: Xapiand: A fast, simple, modern search and storage engine

#19

Post by hardwaresofton »

mdnormy wrote: post #13
cetra3 wrote: post #7

Earlier quoted context omitted.

Also worth mentioning is Toshi: https://github.com/toshi-search/Toshi Toshi is to ElasticSearch as Tantivy is to Lucene if that makes sense. Obviously as they are new they are not at feature parity, but Tantivy does win at some benchmarks: https://tantivy-search.github.io/bench/

Non-native English speaker. But isn't easier to understand like so, "Tantivity to Toshi, is as Lucene to Elasticsearch"

As a native english speaker, the earlier phrase ("tantivy is to toshi as lucene is to elastic search") is easier for me to understand. I find your phrase a bit harder to understand, but it looks like just the kind of reorganization other languages do structure wise -- I don't know how to express it in proper grammatical terms, but the way the prepositions are swapped around makes it seem like native english words but with a non-english structure.

It might have to do with the use of Analogy questions in the SAT (a standardized test all but required for high school students wanting to attend good colleges in America), though it looks like they've been removed?[0].

"_____ is to ___ as ____ is to ______" was the verbatim format of those test questions.

[0]: https://blog.prepscholar.com/sat-analogies-and-comparisons-w...

Post reply on HN