Viewing profile — PSeitz

Hi, tantivy dev here. There are two recent performance improvements in tantivy, which should make term aggregations considerable faster. https://github.com/quickwit-oss/tantivy/pul…

Also we are working on some performance improvements based on the benchmark comparison, as they highlighted some areas we can improve in tantivy.

tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only th…

You can just put everything in a JSON field in tantivy and set it to INDEXED and FAST

They serve quite different use cases. quickwit was built to handle extremely large data volumes, you can ingest and search TB and PB of logs. meilisearches indexing doesn't scale a…

> Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config... Yes that's how you use tantivy normally, not sure which json…

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

>Roaring bitmaps and similar data structures get their speed from decoding together consecutive groups of elements, so if you do sequential decoding or decode a large fraction of t…

> It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps), PEF would likely be faste…

Indeed if you are colorblind that may be an issue. I'll address it If I get to it. Otherwise it assigns names to colors, no idea how you could be confused by that.

file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.

As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s

I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex

But I like coal, so coal can't be the problem either. ROLF!

Or with einstein "yeah you were pretty much right. we don't have anything new"

The performance comparisons between gc and non-gc languages show otherwise. Also, memory consumption eventually costs cpu cycles.

Garbage collector is actually something you'd want, if it caused no performance hit.

Some article about development a approach is not news. Also, hackernews is largely composed of non-news information, else I wouldn't be here ;)

I use a state machine for text https://github.com/BurntSushi/fst

But it does work in Tokyo, maybe it's the people

Internet availability is not a problem. Just check cable availability and your up and running with 100Mbits+ in a few days.