Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch?
I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.
ClickHouse as an alternative to Elasticsearch for log storage and analysis
21–30 of 140 posts
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#22I don't have any production experience running Clickhouse, but I have used it on a side project for an OLAP workload. Compared to Postgres Clickhouse was a couple orders of magnitude faster (for the query pattern), and it was pretty easy to setup a single node configuration compared to lots of the "big data" stuff. Clickhouse is really a game changer.
what happens when your data doesn't fit in a single node anymore?
There's replication in ClickHouse and you can just shove reads off to one of them if you'd like. From a backup/safety standpoint that's important, but I think there are other options besides just replicas, of course.
From an operations standpoint, however, ClickHouse is ridiculously efficient at what it does. You can store tens of billions, probably trillions of records on a single node machine. You can query at tens of billions of rows a second, etc, all with SQL. (The only competitor I know of in the same class is MemSQL.) So another thing to keep in mind is you'll be able to go much further with a single node using ClickHouse than the alternatives. For OLAP style workloads, it's well worth investigating.
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#23Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.
https://typesense.org/ comes to mind. Has support for Algolia's instantsearch.js as well.
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#24Sentry.io is using ClickHouse for search, with an API they built on top of it to make it easier to transition if need be. They blogged about it at the time they adopted it:
https://blog.sentry.io/2019/05/16/introducing-snuba-sentrys-...
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#25My biggest problem with Elasticsearch is how easy it is to get data in there and think everything is just fine... until it falls flat on its face the moment you hit some random use case that, according to Murphy's law, will also be a very important one. I wish Elasticsearch were maybe a little more opinionated in its defaults. In some ways Clickhouse feels like they filled the gap not having opinionated defaults crea…
Would you care to elaborate on what happened in your case. My company is using ElasticSearch extensively, and it is mission critical for us. I fear something might happen one day
It's been a few years so the details are fuzzy, but iirc it was just simple things like index sizing, managing shard count, some JVM tuning, certain aggregated fields blowing up once we had more data in our instances...
We also got our data very out of order. We had embedded devices logging analytics that would phone home very infrequently, think months between check-ins
I forget why but that became a big issue at some point, bringing the instance to its knees when a few devices started to phone-in covering large periods of time.
ES just has a ton of knobs, I imagine if its been important to you, you have people specializing in keeping it running, which is great... but the amount of complexity there that is specific to ES is really high.
It's not like there's no such thing as a Postgres expert for example, but you don't need to hire a Postgres wizard until you're pretty far in the weeds. But I feel like you should have an ES wizard to use ES, which is a little unfortunate
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#26I'm happy that more people are "discovering" ClickHouse.
ClickHouse is an outstanding product, with great capabilities that serve a wide array of big data use cases.
It's simple to deploy, simple to operate, simple to ingest large amounts of data, simple to scale, and simple to query.
We've been using ClickHouse to handle 100's of TB of data for workloads that require ranking on multi-dimensional timeseries aggregations, and we can resolve most complex queries in less than 500ms under load.
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#27ClickHouse is incredible. It has also replaced a large, expensive and slow Elasticsearch cluster at Contentsquare. We are actually starting an internal team to improve it and upstream patches, email me if interested!
Can you share some more details?
How many nodes on both? How much data ingested and stored? What’s the query load?
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#28I've been recording a podcast with Commercial Open Source company founders (Plug! https://www.flagsmith.com/podcast) and have been surprised how often Clickhouse has come up. It is always referred to with glowing praise/couldn't have built our business without it etc etc etc.
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#29A related database using ideas from Clickhouse: https://github.com/VictoriaMetrics/VictoriaMetrics
Are you familiar with VictoriaMetrics?
Can you elaborate on how it is similar and dissimilar to Clickhouse?
What specific techniques are the same?
Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis
#30Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.
Not sure if it's "new" but it's always interesting: Postgres offers a fine set of full-text search functionality, with the advantage of being able to also use the other ways in which Postgres shines: document-like data (storing JSON in columns), horizontal replication, PostGIS, and so on.