Quickwit 0.8: Indexing and Search at Petabyte Scale
1–10 of 31 posts
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#2Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage.
I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by embedding it in Elixir, and the API and docs have been quite pleasant to work with. Hats of to the team, looking forward to what's coming next!
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#313.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#413.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.
Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#5Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…
Some companies are using it with AWS Lambda to scale to 0.
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#6[deleted]
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#713.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.
What is your frame of reference?
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#8Amazing to see how far Tantiviy has come. Remember using and making some smaller contributions to this 3 years ago - slop to phrase queries for example. Curious how the design has changed to enable large scale production usage.
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#9Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by e…
Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?
Re: Quickwit 0.8: Indexing and Search at Petabyte Scale
#1013.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.
Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".
I never being able to understand why log indexing has to build inverted index. Decent columnar store with partitioning by date should be enough to quickly filter gigabytes of logs.