ClickHouse: An open-source column-oriented database management system • Hacker News Board

Re: ClickHouse: An open-source column-oriented database management system

#21

Post by wiredfool » Fri, May 28, 2021, 10:18 AM UTC

momothereal wrote: post #13

People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…

I ran across this recently: https://eng.uber.com/logging/

Uber's use of clickhouse to handle semi-structured data from logging.

Re: ClickHouse: An open-source column-oriented database management system

#22

Post by tototom » Fri, May 28, 2021, 10:29 AM UTC

pibi wrote: post #15

Great software. We are managing terabytes of stocks data and realtime market scanners queries across all market (billions of books and timesales) with hundreds of concurrent requests. We were using kdb before, but clickhouse is more scalable, way cheaper and much more easy to grasp for a newbie.

How much data? And which types of application/apis are connected to CH?

Re: ClickHouse: An open-source column-oriented database management system

#23

Post by polote » Fri, May 28, 2021, 10:36 AM UTC

des1nderlase wrote: post #9

coolspot wrote: post #6

Earlier quoted context omitted.

> tosh submits 10-20 articles a day That’s kinda sad

I didn't know HN allows someone to posts that much. This would be a full-time job for a person. Hidden motives?

This is a common pattern. I wouldnt be surprise that 20% of posted links are from the same 10 users

Re: ClickHouse: An open-source column-oriented database management system

#24

Post by twic » Fri, May 28, 2021, 10:39 AM UTC

des1nderlase wrote: post #9

coolspot wrote: post #6

Earlier quoted context omitted.

> tosh submits 10-20 articles a day That’s kinda sad

I didn't know HN allows someone to posts that much. This would be a full-time job for a person. Hidden motives?

Maybe they've got a "submit to HN" bookmarklet, and just whack it whenever they come across something interesting.

Re: ClickHouse: An open-source column-oriented database management system

#25

Post by hocuspocus » Fri, May 28, 2021, 10:47 AM UTC

My team is (over|ab)using Elasticsearch and I've had my eye on ClickHouse for a while. However we're going to migrate everything to AWS and I wonder if RedShift could be a good alternative too, since it's now supporting JSON and semi-structured data apparently.

Re: ClickHouse: An open-source column-oriented database management system

#26

Post by anhldbk » Fri, May 28, 2021, 11:02 AM UTC

momothereal wrote: post #13

People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…

How about using CDC with Apache Kafka?

Re: ClickHouse: An open-source column-oriented database management system

#27

Post by qeternity » Fri, May 28, 2021, 11:06 AM UTC

pibi wrote: post #15

Great software. We are managing terabytes of stocks data and realtime market scanners queries across all market (billions of books and timesales) with hundreds of concurrent requests. We were using kdb before, but clickhouse is more scalable, way cheaper and much more easy to grasp for a newbie.

You’re ingesting real time data into Clickhouse?

Re: ClickHouse: An open-source column-oriented database management system

#28

Post by golondon » Fri, May 28, 2021, 11:09 AM UTC

Is there anyone using it in a schemaless way as in you provide a SaaS service and your customers can send you JSON with various different fields? As far as I remember Sentry uses it this way and people leverage arrays for such things?

Re: ClickHouse: An open-source column-oriented database management system

#29

Post by bayan1234 » Fri, May 28, 2021, 11:10 AM UTC

hocuspocus wrote: post #25

My team is (over|ab)using Elasticsearch and I've had my eye on ClickHouse for a while. However we're going to migrate everything to AWS and I wonder if RedShift could be a good alternative too, since it's now supporting JSON and semi-structured data apparently.

Having used Redshift considerably in the past and experienced significant pain with many aspects of it, I would stay away. There are other much better options, ideally Snowflake.

Re: ClickHouse: An open-source column-oriented database management system

#30

Post by hashhar » Fri, May 28, 2021, 11:20 AM UTC

momothereal wrote: post #13

People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…

Take a look at query engines like Trino (formerly PrestoSQL) [https://trino.io/]. (Disclaimer: I'm a contributor to Trino).

I used it at a previous job to combine data from MongoDB, Kafka, S3 and Postgres to great effect. It tries to push-down as many operations as possible to the source too to improve performance.

Full ANSI SQL support over multiple number of backends (Kafka, Cassandra, Postgres, ClickHouse, S3 and many more).

The best part is it has a plugin ecosystem so you can very easily implement your own connectors and all the heavy lifting gets done by the core-engine while your plugin only has to abstract your backend to concepts that the engine can understand.