Viewing profile — guilload

Quickwit ( https://quickwit.io/ ) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer …

We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.

There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out ( https://commoncrawl.org/2021/04/april-2021-…

We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification…

What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy…