Live data from Hacker News

ClickHouse as an alternative to Elasticsearch for log storage and analysis

pixeljets.com

121–130 of 140 posts

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#121

Earlier quoted context omitted.

Could you share more details about the limited JOIN capabilities? AFAIK, Clickhouse has multiple join algorithms and supports on-disk joins to avoid out of memory: https://github.com/ClickHouse/ClickHouse/issues/10830 https://github.com/ClickHouse/ClickHouse/issues/9702#issueco...

Maybe I’m not doing anything particularly challenging with it, but I’ve not found anything lacking with the join functionality.

One issue I've come across is that the query optimizer in Clickhouse does not propagate `where` clauses through a join. My terminology might be wrong, so consider this example:

select * from a inner join b using (id) where b.foo = 'bar'

Clickhouse will not evaluate `foo = 'bar'` before performing the join, so you might wind up with a join that produces a large intermediate result before the filtering happens. Postgres (probably other databases) will optimize this for you. To force Clickhouse to filter first, you would need to write something like

select * from a inner join ( select * from b where foo = 'bar' ) b using (id)

Maybe not a strict limitation, but the workaround is a bit janky.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#122

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Can clickhouse deal with medium-large blob data? Say the size of a normal email? We are using Postgres to store email at my app: https://hanami.run The log is append only and getting scrub daily. Can clickhouse deal with that? The query is very simple, just need to match exactly a single column(domain) and pagination?

Don’t know how you feel about vendor lock in, but I use Dynamo to store many millions of email logs and it works great. Got tired of having that data in MySQL.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#124

Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.

I'm personally very fond of sonic [0] for full text search. > Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases. It is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query.... > When reviewing Elasticsearch (ELS) and others, we found those were full-fea…

>It is used to index half a billion objects on a $5/mth 1-vCPU SSD cloud server (as of 2019).

Damn. Thank you! This looks very nice!

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#125

Anyone know more lightweight alternative to (ELK) Elastic Stack? I found https://vector.dev but it seems to be only the "L" part.

What about https://github.com/meilisearch/MeiliSearch ?

I am looking into this. Do you have experience with it?

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#126

Earlier quoted context omitted.

I think the author addresses your point one in the article: > SQL is a perfect language for analytics. I love SQL query language and SQL schema is a perfect example of boring tech that I recommend to use as a source of truth for all the data in 99% of projects: if the project code is not perfect, you can improve it relatively easily if your database state is strongly structured. If your database state is a huge JSON…

I disagree with the author on that. Yes, SQL is nicer for structured queries, sure (“KQL” in Kibana is sort of a baby step into querying data stored in Elastic). But in Kibana, I can just type in (for example) a filename, and it will return any result row where that filename is part of any column of data. Also, if I need more structured results (for example, HTTP responses by an API grouped per hour per count), I can…

Agreed here. More and more data is semi structured and can benefit from ES (or mongo) making it easily exploitable. It's a big part of why logstash and elastic came to be.

One of the most beautiful use cases I've ever seen for elasticsearch was custom nginx access log format in json (with nearly every possible field you could want), logged directly over the network (syslogd in nginx over udp) to a fluentd server setup to parse that json + host and timestamp details before bulk inserting to elastic.

You could spin up any nginx container or vm with those two config lines and every request would flow over the network (no disk writes needed!) and get logged centrally with the hostname automatically tagged. It was doing 40k req/s on a single fluentd instance when I saw it last and you could query/filter every http request in the last day (3+bn records...) in realtime.

Reach out to datadog and ask how much they would charge for 100bn log requests per month.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#127
post #118

Almost nobody wants to use elasticsearch. People want to use kibana and put up with elasticsearch.

+1 we actually looked a ch as our debug logs backend and while it is great for the most part (it’s also incredibly memory hungry) kibana is really an es killer feature

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#128

Earlier quoted context omitted.

Yup, reading that comment all I thought was exactly what I said in another comment here, it'll work great until it doesn't, and by then you'll suffer a lot to work around it Same with scaling, scaling ES is super easy until you realize your index sizes aren't playing nicely with sharding or something and have to start working around that. Clickhole feels like it's targeting what most people end up using ES for. Compa…

I manage a fairly small ES cluster of 20 i3en.2xlarge instances that ingest data from 300+ apps. Yes, the only problem I see is the field type collision and it happens occasionally. Otherwise elastic doesn't require much operational time, may be an hour a week. You pretty much want to keep your indices around 50gb and the ILM works well to manage that.

Anyone accepting freeform objects into a single ES index knows the pain of field type collisions

But, most of the time, it "just works".

Hearing these argument about rigid schemas saving time tells me that nobody has had to support teams with 200+ apps.

^ This guy actually manages infra

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#129

Earlier quoted context omitted.

Maybe I’m not doing anything particularly challenging with it, but I’ve not found anything lacking with the join functionality.

One issue I've come across is that the query optimizer in Clickhouse does not propagate `where` clauses through a join. My terminology might be wrong, so consider this example: select * from a inner join b using (id) where b.foo = 'bar' Clickhouse will not evaluate `foo = 'bar'` before performing the join, so you might wind up with a join that produces a large intermediate result before the filtering happens. Postgre…

Interesting, I swear I’ve done that before and has it respect the clause.

You can also use PREWHERE I believe, which has the benefit of increasing performance.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#130
post #32

I think it's an unfair comparison, notably because: 1) Clickhouse is rigid-schema + append-only - you can't simply dump semi-structured data (csv/json/documents) into it and worry about schema (index definition) + querying later. The only clickhouse integration I've seen up close had a lot of "json" blobs in it as a workaround, which cannot be queried with the same ease as in ES. 2) Clickhouse scalability is not as s…

> you can't simply dump semi-structured data (csv/json/documents) into it and worry about schema (index definition) + querying later Unless you love rewrites, you can't simply dump semi-structured data into ElasticSearch either. Seen multiple apps with 5x or worse ES storage usage tied to 'data model' or lack thereof, and fixing it inevitably means revisiting every piece of code pushing stuff into and out of ES. I lo…

Imagine trying to make the argument that forcing your developers/clients to send all their telemetry on fixed/rigid schemas to make it immediately queryable is quicker than updating 1 line of an etl script on the data warehouse side. That adding a new queryable field to your event now requires creating migration scripts for the databases and api versioning for the services so things don't break and old clients can continue using the old schema. Imagine making a central telemetry receiver that needs to support 200+ different external apps/clients, with most under active development - adding new events and extending existing ones - being released several times per day. What's the alternative you're proposing? Just put it in a json column and make extractors in the databases every time you want to analyze a field? I've seen this design pattern often enough in MSSQL servers with stored procedures... Talk to me about painful rewrites.

I'll take semi-structured events parsed and indexed by default during ingestion over flat logs + rigid schema events any day. When you force developers to log into a rigid schema you get json blob fields or "extra2" db fields, or perhaps the worst of all, no data at all since it's such a pain in the ass to instrument new events.

We're talking about sending, logging and accessing telemetry. The goal is to "see it" and make it accessible for simple querying and analysis - in realtime ideally, and without a ticket to data engineering.

ES type-inferrence and wide/open schema with blind json input is second to none as far as simplicity of getting data indexed goes. There are tradeoffs with the defaults such as putting lots of text that you don't need to fulltext search - you might want to tell ES that it doesn't need to parse every word into an index if you don't want to burn extra cpu and storage for nothing. This is one line of config at the cluster level and can be changed seamlessly while running and ingesting data.

I guarantee you there is more semi-structured data in the world than rigid schema, and for one simple reason: It's quicker to generate. The only argument against it has thus far been "yeah but then it's difficult to parse and make it queryable again" and suddenly you've come full circle and you have the reason elasticsearch exists and shines (extended further on both ends by logstash and kibana).

I'm not saying it makes sense to do away with schemas everywhere but for logging and telemetry - of any that you actually care to analyze anyway - there is rarely a reason to go rigid schema on the accepting or processing side since you'll be working with, in the vast majority of cases, semi-structured data.

Changing ES index mappings on the fly is trivial, you can do it with an much ease as alter table on clickhouse, and you have the luxury of doing it optimistically and after the fact, once your data/schema has stabilized.

Rewriting the app to accommodate this should never be required unless you really don't know how to use indexing and index mapping in ES. You would, however, have to make changes to your app and/or database and/or ETL every time you wanted to add a new queryable field to your rigid-schema masterpiece.

Ultimately, applications have always and will always generate more data than will be ultimately analyzed so saving development time on that generating end (by accepting any semi-structured data without first having to define a rigid schema) is more valuable than saving it on the end that is parsing a subset of that data. Having to involve a data team to deploy an alter table so you can query a field from your json doesn't sound like the hallmark of agile self-serve. I also believe strongly and fundamentally that encouraging product teams and their developers to send and analyze as much telemetry as both their hearts desire and DPOs agree to without worrying about the relatively trivial cost of parsing and storing it, will always come out on top vs creating operational complexity over the same. Maybe if you have a small team logging billions of heavy, never-changing events will seldom get queried it would tip the scales in favor of using rigid schema. I counter: you don't need telemetry you need archiving.

On that subject of pure compute and storage/transfer efficiency: Yes both rigid schema and processing-by-exception will win here every time as far as cycles and bits go. Rarely is the inefficiency of semi-structured so high that it merits handicapping an entire engineering org into dancing around rigid schemas to get their telemetry accepted and into a dashboard.

I hear you, platform ops teams... "But the developers will send big events! ! There will be lots of data that we're parsing and indexing for nothing!" Ok - so add a provision to selectively ignore those? Maybe tell the offender to stop doing it? On the rare occasion that this happens (I've seen 1 or 2 events out of 100s in my anecdotal experience) you may require some human intervention. Compare this labor requirement to the proposed system where human intervention is required every time somebody wants to look at their fancy new field.

In practice, I've not seen it be a nightmare unless you've got some very bad best practices on the ingestion or indexing side - both of which are easily remedied without changing much if anything outside of ES.

I think clickhouse is pretty cool, but it's not handing anywhere near the constraints that ES does even without logstash and kibana. ES is also getting faster and more efficient at ingestion/parsing with every release - releases the seem to be coming faster and faster these days.

Post reply on HN