Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse as an alternative to Elasticsearch for log storage and analysis

pixeljets.com

Post reply on HN
Print view View original

101–110 of 140 posts

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#101

Post by jcims »

supergirl wrote: post #91
jcims wrote: post #81

Does ClickHouse or anything else out there that even remotely compete with Splunk for adhoc troubleshooting/forensics/threat hunting type work? I started off with Splunk and every time I try Elasticsearch I feel like I'm stuck in a cage. Probably why they can charge so much for it.

why is splunk better than ES?

I really want to answer you but I'm struggling a bit b/c I haven't worked with ES in a minute. Splunk just tends to be able to eat just about any kind of structured or unstructed content, operates on a pipeline concept similar to that of a unix shell or (gasp) powershell, and has a rich set of data manipulation of modification commands built in:

https://docs.splunk.com/Documentation/SplunkLight/7.3.6/Refe...

I primarily use it for security-related analysis, which is lowish on metrics and high on adhoc folding and mutilation of a very diverse set of data structures and types.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#102

Post by pachico »

I've been using it successfully in production for year and a half. I can think of no other database that would give me real time aggregation over hundreds of millions of rows inserter every day for virtually zero cost. It's just a marvelous work.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#104

Post by yamrzou »

the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Could you share more details about the limited JOIN capabilities? AFAIK, Clickhouse has multiple join algorithms and supports on-disk joins to avoid out of memory:

https://github.com/ClickHouse/ClickHouse/issues/10830

https://github.com/ClickHouse/ClickHouse/issues/9702#issueco...

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#105

Post by free652 »

BoorishBears wrote: post #76
anaphor wrote: post #42

Earlier quoted context omitted.

You might be able to just put whatever you want into an Elasticsearch index, but I wouldn't recommend doing that. It could severely limit how you can query your data later, see: https://www.elastic.co/guide/en/elasticsearch/reference/curr... Also it can cause performance problems if you have really heterogeneous data with lots of different fields https://www.elastic.co/guide/en/elasticsearch/reference/curr...

Yup, reading that comment all I thought was exactly what I said in another comment here, it'll work great until it doesn't, and by then you'll suffer a lot to work around it Same with scaling, scaling ES is super easy until you realize your index sizes aren't playing nicely with sharding or something and have to start working around that. Clickhole feels like it's targeting what most people end up using ES for. Compa…

I manage a fairly small ES cluster of 20 i3en.2xlarge instances that ingest data from 300+ apps. Yes, the only problem I see is the field type collision and it happens occasionally.

Otherwise elastic doesn't require much operational time, may be an hour a week.

You pretty much want to keep your indices around 50gb and the ILM works well to manage that.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#106

Post by mgachka »

jetter wrote: post #36
sylvinus wrote: post #6

ClickHouse is incredible. It has also replaced a large, expensive and slow Elasticsearch cluster at Contentsquare. We are actually starting an internal team to improve it and upstream patches, email me if interested!

Yep, do you guys have a writeup on this? Altinity actually mention Contentsquare case in their video, here: https://www.youtube.com/watch?t=2479&v=pZkKsfr8n3M&feature=y...

Hi

I'm of the guy who did the 2 presentations of Clickhouse at ContentSquare. There are no blog posts on the migration from ES to CH. But you can find the slides of the 2018 presentation here https://www.slideshare.net/VianneyFOUCAULT/clickhouse-meetup... And the slides of the 2019 presentation here https://www.slideshare.net/VianneyFOUCAULT/meetup-a-successf...

There is also a video recording of the 2019 presentation available here. https://www.youtube.com/watch?v=lwYSYMwpJOU nb: The video is not great because the camera is often losing focus but it's still understandable.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#107

Post by mgachka »

polote wrote: post #49
jetter wrote: post #36

Earlier quoted context omitted.

Yep, do you guys have a writeup on this? Altinity actually mention Contentsquare case in their video, here: https://www.youtube.com/watch?t=2479&v=pZkKsfr8n3M&feature=y...

I'm not sure there is a public writeup. I know that the incredibly talented guy who created the first CH setup at CS planned to write a more global post about data analytics at scale, but after 2 years I still wait for it

I stopped answering to people about the release date of my next blog post because I'm always postponing it ;-).

But don't worry Paul, the day I'll release it you'll be one of the first to be informed.

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#109

Post by mgachka »

dominotw wrote: post #15

How does clickhouse compare to druid, pinot, rockset (commercial), memsql (commercial). I know clickhouse is easier to deploy. But from user's perspective is clickhouse superior to the others?

FYI, we're using clickhouse since 2018 at ContentSquare.

I did a few POCs to compare clickhouse vs other databases on ContentSquare's use case. One of them was memSQL. Although memSQL was very good, since we don't need to JOIN big datasets or need killer features like fulltext search, clickhouse gave a better perf/cost ratio for us (I don't remember exactly but it was at least twice cheaper).

Re: ClickHouse as an alternative to Elasticsearch for log storage and analysis

#110

Post by FridgeSeal »

the-alchemist wrote: post #97

Also wanted to share my overall positive experience with Clickhouse. UPSIDES * started a 3-node cluster using the official Docker images super quickly * ingested billions of rows super fast * great compression (of course, depends on your data's characteristics) * features like https://clickhouse.tech/docs/en/engines/table-engines/merget... are amazing to see * ODBC support. I initially said "Who uses that??", but we…

Most minor of nitpicks:

> timeseries-oriented database

Technically it’s a column oriented database that is good at time series stuff. I only say that because I know there are some databases that are even more specialised towards timeseries and ClickHouse can do way more.

Post reply on HN