Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quickwit 0.8: Indexing and Search at Petabyte Scale

quickwit.io

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#21

Post by bbkane »

tecleandor wrote: post #9
dracyr wrote: post #2

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…

Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?

I use OpenObserve and I quite enjoy it

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#22

Post by mdaniel »

tecleandor wrote: post #9
dracyr wrote: post #2

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…

Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?

in case it matters to others, https://github.com/openobserve/openobserve/tree/v0.7.0 is the last Apache2 licensed copy before they went AGPL with 0.7.1

https://github.com/openobserve/openobserve/blob/v0.7.0/.env.... is some "onoz" for me, but just recently someone submitted https://github.com/aenix-io/etcd-operator to the CNCF sandbox so maybe things have gotten better around keeping that PoS alive

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#23

Post by dist1ll »

fulmicoton wrote: post #18
dist1ll wrote: post #13

Earlier quoted context omitted.

Per-core store bandwidth is at least 14GB/s on Zen3, 35GB/s for non-temporal stores. Parsing JSON can be done at +2GB/s. It's very healthy to take maximum bandwidth limits into consideration when reasoning about performance. For instance, for temporal stores, the bottlenecks you see are due to RAM latency and memory parallelism, because of the write-allocate. The load/store uarch can actually retire way more data fro…

What we do is CPU bound and we are not just parsing JSON here. The largest work we do is building an inverted index. Oversimplified, it is equivalent to this: inverted_index = defaultdict(list) for (doc_id, doc_json) in enumerate(doc_jsons): c = json.loads(payload) for (field, field_text) in c.items(): for (position, token) in enumerate(): inverted_index[token].push((doc, position)) serialize_in_compressed_way_that_a…

I'm curious, what is your frame of reference with regards to maximum speed of building inverted indices? Like, what is the maximum throughput you'd expect for this type of task, and what is your reasoning for it?

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#26

Post by nextaccountic »

dracyr wrote: post #2

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…

Tantivity is great!

Here is a postgres extension that uses it to provide full text search

https://blog.paradedb.com/pages/introducing_bm25

https://news.ycombinator.com/item?id=37557127

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#27

Post by francoismassot »

nextaccountic wrote: post #26
dracyr wrote: post #2

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…

Tantivity is great! Here is a postgres extension that uses it to provide full text search https://blog.paradedb.com/pages/introducing_bm25 https://news.ycombinator.com/item?id=37557127

tantivy, not tantivity!!!!!

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#28

Post by lrx »

godber wrote: post #19

We did some experimentation with quickwit about a year ago, writing about 1m docs/second of data into it for several months. It worked well and was pretty straight forward to learn and operate. If we didn’t also manage our own S3/Ceph it might be a big win, once feature complete. It’s definitely worth a look.

I think you can use quickwit with a self-hosted S3-compatible object store.

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#29

Post by kikimora »

fulmicoton wrote: post #16
kikimora wrote: post #10

Earlier quoted context omitted.

I never being able to understand why log indexing has to build inverted index. Decent columnar store with partitioning by date should be enough to quickly filter gigabytes of logs.

Quickwit co-founder here... I actually agree. For a few GBs, done right, columnar works fine AND is cost efficient. After all, it does not matter much if a log search query answers in 300ms or 1s. However, there are use cases where a few GB just does not cut it. The tale saying that you can always prune your dataset using timestamp and tags is simply not always valid.

Can you share your experience of when columnar fails?

It is possible to scan NVMe at a speed of multiple GB/sec, scans can be parallel and happen on multiple disks, over compressed data (10 Gb of logs ~ 1Gb to scan), data can be segmented and prefaced with Blum filters, to quickly check if a segment is worth scanning.

Re: Quickwit 0.8: Indexing and Search at Petabyte Scale

#30

Post by nh2 »

kikimora wrote: post #29
fulmicoton wrote: post #16

Earlier quoted context omitted.

Quickwit co-founder here... I actually agree. For a few GBs, done right, columnar works fine AND is cost efficient. After all, it does not matter much if a log search query answers in 300ms or 1s. However, there are use cases where a few GB just does not cut it. The tale saying that you can always prune your dataset using timestamp and tags is simply not always valid.

Can you share your experience of when columnar fails? It is possible to scan NVMe at a speed of multiple GB/sec, scans can be parallel and happen on multiple disks, over compressed data (10 Gb of logs ~ 1Gb to scan), data can be segmented and prefaced with Blum filters, to quickly check if a segment is worth scanning.

I'm not the person you asked, but say you have 10 TB of logs.

Assuming 3 GB/s SSD, 10 SSDs, and a compression as you suggested of 10x, a query for finding a string in the text would take 10000 / 3 / 10 / 10 = 33 seconds.

With an index, you can easily get it 100x faster, and that factor gets larger as your data grows.

In general it's just that O(log(n)) wins over O(n) when n gets large.

I didn't take your Bloom filter idea into consideration as it is not immediately obvious how a Bloom filter can support all filter operations that an index can. Also, the index gives you the exact position of the match, when the bloom filter only gives you existence, thus potentially still resulting in a large read amplication factor of a scan in the segment vs direct random access.

Post reply on HN