Live data from Hacker News

ClickHouse as an alternative to Elasticsearch for log storage and analysis

pixeljets.com

131–140 of 140 posts

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#131
post #125

Earlier quoted context omitted.

What about https://github.com/meilisearch/MeiliSearch ?

I am looking into this. Do you have experience with it?

It is good. I can't find any CDC for Postgres for the incremental sync. And so I had to use the bulk update/sync and that causes performance issues occasionally. Also, some Algolia features are not available yet https://github.com/meilisearch/instant-meilisearch/issues/21...

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#132

Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.

FWIW...

1. Toshi https://github.com/toshi-search/Toshi (Rust, 3.1k stars)

2. Tantivy https://github.com/tantivy-search/tantivy (Rust, 4.5k stars)

3. PISA https://github.com/pisa-engine/pisa (C++, 486 stars)

4. Bleve https://github.com/blevesearch/bleve (Go, 7.4k stars)

5. Sonic https://github.com/valeriansaliou/sonic (Rust, 10.9k stars)

6. Partial comparison https://mosuka.github.io/search-benchmark-game/ (tantivy Vs lucene Vs pisa Vs bleve)

7. Bayard https://github.com/bayard-search/bayard (Rust, on top of Tantivy, 1.4k stars)

8. Blast https://github.com/mosuka/blast (Go, on top of Bleve, 930 stars)

Algolia alternatives with some compatibility

1. MeiliSearch https://github.com/meilisearch/MeiliSearch (Rust, 12.4k stars)

2. typesense https://github.com/typesense/typesense (C++, 5.1k stars)

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#133

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Most minor of nitpicks: > timeseries-oriented database Technically it’s a column oriented database that is good at time series stuff. I only say that because I know there are some databases that are even more specialised towards timeseries and ClickHouse can do way more.

I'm working on VictoriaMetrics - a fast specialized time series database and monitoring solution built on top of ClickHouse architecture ideas. [1]

[1] https://valyala.medium.com/how-victoriametrics-makes-instant...

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#135
post #48

Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.

https://github.com/meilisearch/MeiliSearch gets a lot of traction recently. There are also Sphinx and its fork https://manticoresearch.com/ - very lightweight and fast.

I immediately thought of Sphinx when I saw MeiliSearch... it's uncanny how the use case and implementation semantics haven't changed much in 15 years.

The beauty of pointing it to your mysql tables and getting fulltext-via-api on the other side was quite nice.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#136

Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.

FWIW... 1. Toshi https://github.com/toshi-search/Toshi (Rust, 3.1k stars) 2. Tantivy https://github.com/tantivy-search/tantivy (Rust, 4.5k stars) 3. PISA https://github.com/pisa-engine/pisa (C++, 486 stars) 4. Bleve https://github.com/blevesearch/bleve (Go, 7.4k stars) 5. Sonic https://github.com/valeriansaliou/sonic (Rust, 10.9k stars) 6. Partial comparison https://mosuka.github.io/search-benchmark-game/ (tantivy Vs…

Wow! Thank you!

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#137
post #24

Sentry.io is using ClickHouse for search, with an API they built on top of it to make it easier to transition if need be. They blogged about it at the time they adopted it: https://blog.sentry.io/2019/05/16/introducing-snuba-sentrys-...

I like sentry but it's the only app that I know uses 25 or so containers to stich itself together to get running which seemed insane, not to mention the usage of much ram.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#138
post #32

I think it's an unfair comparison, notably because: 1) Clickhouse is rigid-schema + append-only - you can't simply dump semi-structured data (csv/json/documents) into it and worry about schema (index definition) + querying later. The only clickhouse integration I've seen up close had a lot of "json" blobs in it as a workaround, which cannot be queried with the same ease as in ES. 2) Clickhouse scalability is not as s…

> 3) Elastic is more than elasticsearch - kibana and the "on top of elasticsearch" featureset is pretty substantial.

Kibana is just messy. Their demos don't show any actionable intelligence but just dumps data in various ways and the interface doesn't look focused. It feels painful to deal with it daily.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#139
post #118

Almost nobody wants to use elasticsearch. People want to use kibana and put up with elasticsearch.

> People want to use kibana and put up with elasticsearch.

I don't buy this.

It's just a mess of data dumps and it's not exactly providing focused experience. You need to take a full month to show what you want comfortably.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#140
post #22

Earlier quoted context omitted.

what happens when your data doesn't fit in a single node anymore?

There's replication in ClickHouse and you can just shove reads off to one of them if you'd like. From a backup/safety standpoint that's important, but I think there are other options besides just replicas, of course. From an operations standpoint, however, ClickHouse is ridiculously efficient at what it does. You can store tens of billions, probably trillions of records on a single node machine. You can query at tens…

I don't know why people still call replication a backup.
Post reply on HN