Viewing profile — fulmicoton

One trouble I could see with your approach is that you treat the information "Doc at pos i" beats "Doc at pos j" independently from i and j. Intuitively, it is not as critical when…

This bug hit us, and yes, I hadn't thought of just switching to opendal. That's indeed a great reminder.

No. Quickwit was founded well before Warpstream and it did not inspire us. The Husky blog post was released after we released a few versions of quickwit if I recall correctly. It w…

Our seed round was 100% made of SAFE, so VCs did not have the power to force us to do anything. The sentence in the blog post is a tad misleading. I suspect François is not really …

Developer of tantivy chiming in! (I hope that's ok) Database performance is a space where there are a lot of lies and bullshit, so you are 100% right to be suspicious. I don't know…

Yes. We should shut down this demo. We reduced the hardware to cut down our costs. Right now it runs a ludicrously small amount of hardware.

Quickwit is targetting logs: - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp. - its does not use an SSD cache. Quickw…

This is NOT about transaction log. This is application logs. The thing you generate via Log4j for instance. Also 100PB is measured as the input format (JSON). Internally Quickwit w…

Security and customer support are the two main reasons why people want a super long retention. Medium retention (1 or 2 months) is still very appreciable if some issue in your bugt…

It is pretty much the same as Lucene. The compression ratio is very specific logs and depends on the logs themselves. (Often it is not that good)

Quickwit (like Elasticsearch/Opensearch) stores you data compressed with ZSTD in a row store, builds a full text search index, and stores some of your fields in a columnar. The "co…

Again, this is application logs. The stuff you would log in your program with log4j for instance. With a microservices architecture in particular that can pile up rapidly.

Thank you for the kind word @ZeroCool2u ! :)

Building an inverted index is actually very cpu intensive. I think we are the fastest on that (if someone knows something faster than tantivy at indexing I am interested). I'd be r…

If you can limit your research to GBs of logs, I kind of agree with you. It's ok if a log search request takes 100ms instead of 2s, and the "grep" approach is more flexible. Usuall…

The data is just Binance's application logs for observability. Typically what a smaller business would simply send to Datadog. This log search infra is handled by two engineers who…

Quickwit is designed to do full-text search efficiently with an index stored on an object storage. There are no equivalent technology, apart maybe: - Chaossearch but it is hard to …

This is their application logs. They need to search into it in a comfortable manner. They went for a search engine with Elasticsearch at first, and Quickwit after that because even…

Thank you @tyler!!!

Thank you so much for sharing!!!

Yes

A math puzzle, its relationship with the average case complexity of computing top-K using a min heap, and a simple algorithm that performs better.

We used to have one. Maybe we can revive it. What is your use case?

What we do is CPU bound and we are not just parsing JSON here. The largest work we do is building an inverted index. Oversimplified, it is equivalent to this: inverted_index = defa…