Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse as an alternative to Elasticsearch for log storage and analysis

pixeljets.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#111

Post by FridgeSeal »

yamrzou wrote: post #104
the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Could you share more details about the limited JOIN capabilities? AFAIK, Clickhouse has multiple join algorithms and supports on-disk joins to avoid out of memory: https://github.com/ClickHouse/ClickHouse/issues/10830 https://github.com/ClickHouse/ClickHouse/issues/9702#issueco...

Maybe I’m not doing anything particularly challenging with it, but I’ve not found anything lacking with the join functionality.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#112

Post by scomp »

djxfade wrote: post #20
BoorishBears wrote: post #16

My biggest problem with Elasticsearch is how easy it is to get data in there and think everything is just fine... until it falls flat on its face the moment you hit some random use case that, according to Murphy's law, will also be a very important one. I wish Elasticsearch were maybe a little more opinionated in its defaults. In some ways Clickhouse feels like they filled the gap not having opinionated defaults crea…

Would you care to elaborate on what happened in your case. My company is using ElasticSearch extensively, and it is mission critical for us. I fear something might happen one day

I work for a company that uses it as a mission critical product. It provides the search in our SAAS. I'd say it mysteriously fails 3-4 times a year. But no one wants to invest or encourage anyone to look at it proactively or even re-actively. We've had JVM issues and indexers spiral out of control using 99%CPU only to stop at random hours later. It's definitely a product to learn before it fails.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#113

Post by IanCal »

the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

I'd like to add an upside which is:

Totally great and simple on a single node.

I looked at a bunch of analytical databases and had a lot that started with "so here's a basic 10 node cluster". Clickhouse installed and worked instantly with decent but not "big" data with no hassle. A hundred million rows with lots of heavy text blobs and a lot of columns, that kind of thing. Happily dealt with triple nested joins over that, and with billions of entries in arrays on those columns didn't bat an eye.

I'm sure I could do some great magic in postgres but naive work didn't give anywhere near the same results as clickhouse (obvious caveat for my workload).

Pretty good with JSON data, my only issue there at the time (may have improved) was you had to format the JSON quite strictly.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#114

Post by pojzon »

free652 wrote: post #105
BoorishBears wrote: post #76

Earlier quoted context omitted.

Yup, reading that comment all I thought was exactly what I said in another comment here, it'll work great until it doesn't, and by then you'll suffer a lot to work around it Same with scaling, scaling ES is super easy until you realize your index sizes aren't playing nicely with sharding or something and have to start working around that. Clickhole feels like it's targeting what most people end up using ES for. Compa…

I manage a fairly small ES cluster of 20 i3en.2xlarge instances that ingest data from 300+ apps. Yes, the only problem I see is the field type collision and it happens occasionally. Otherwise elastic doesn't require much operational time, may be an hour a week. You pretty much want to keep your indices around 50gb and the ILM works well to manage that.

What about threadwriterejects and max-number-of-shards. If you don’t take into consideration how much data you ingest and in what format it should be afterwards and monitor that constantly:

“You gonna have a bad time”. You can automate a lot of stuff around elasticsearch, but when you provide/source it within company - other teams may not be as knowlegable and can shoot themselves into the foot very easly.

Ive seen it multiple times by now. People have no idea how to manage the size of their clusters.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#115

Post by FridgeSeal »

wiradikusuma wrote: post #4

Anyone know more lightweight alternative to (ELK) Elastic Stack? I found https://vector.dev but it seems to be only the "L" part.

ClickHouse will happily replace the ElasticSearch bit, and there’s a few open source dashboards you could use as a kibana stand in: - Metabase (with ClickHouse plug-in) - Superset - Grafana

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#116

Post by kristjansson »

jerf wrote: post #93
phillc73 wrote: post #66

Earlier quoted context omitted.

I agree that dbplyr is a nice way to query databases, if already familiar with dplyr (actually I think dtplyr is more interesting for operating on data.tables). However, I'm not sure I really understand your point about the "if" statement. If the data is already in a dataframe, why not still use the "if" statement, but one of the packages I mentioned earlier to further modify the data? E.g. if (x = 10) { duckdf("SELE…

I read down the thread as it stands now, and the basic problem is this. Write the following function: Compose("SELECT a, b.revenue, b.name FROM table INNER JOIN b ON whatever WHERE name LIKE 'b%'", "revenue > 10000") to yield a new valid SQL statement that composes these two fragments together in the way that it is obvious that I want. You basically can't. If you can in this small example, I can easily complexify the…

SQL is a good language, but a terrible API.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#117

Post by kureikain »

the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Can clickhouse deal with medium-large blob data? Say the size of a normal email?

We are using Postgres to store email at my app: https://hanami.run The log is append only and getting scrub daily.

Can clickhouse deal with that? The query is very simple, just need to match exactly a single column(domain) and pagination?

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#119

Post by gravypod »

moralestapia wrote: post #21

Sorry to hijack the thread but can anyone suggest alternatives to the 'search' side of Elasticsearch? I haven't been following the topic and there's probably new and interesting developments like ClickHouse is for logging.

I'm personally very fond of sonic [0] for full text search.

> Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases. It is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query....

> When reviewing Elasticsearch (ELS) and others, we found those were full-featured heavyweight systems that did not scale well with Crisp's freemium-based cost structure.

> At the end, we decided to build our own search backend, designed to be simple and lightweight on resources

[0] - https://github.com/valeriansaliou/sonic

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#120

Post by u678u »

IanCal wrote: post #113
the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

I'd like to add an upside which is: Totally great and simple on a single node. I looked at a bunch of analytical databases and had a lot that started with "so here's a basic 10 node cluster". Clickhouse installed and worked instantly with decent but not "big" data with no hassle. A hundred million rows with lots of heavy text blobs and a lot of columns, that kind of thing. Happily dealt with triple nested joins over…

Thanks, this is just what I'm looking for.
Post reply on HN