Clickhouse is easily my favorite secret weapon.
Lowered our costs by ~50% and reduced data processing times by orders of magnitudes.
ClickHouse: An open-source column-oriented database management system
31–40 of 58 posts
Re: ClickHouse: An open-source column-oriented database management system
#32People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
Don't know about CH, but with apache druid, you can "join" with a rdbms query via a LOOKUP sql function.
Re: ClickHouse: An open-source column-oriented database management system
#33I'm not sure why this got posted again on HN. It's been posted numerous times, though in the past the url used to be https://github.com/yandex/ClickHouse so it wouldn't have been caught as a duplicate. Just searching on HB algolia there are numerous posts. eg: https://hn.algolia.com/?q=clickhouse
tosh submits 10-20 articles a day (yet averages about 1 comment every 2-3 days). There's bound to be some dupes.
FWIW, I don't see a problem with that level of posting. They are simply sharing things they find interesting with the community. If the community disagrees, it does not get upvotes. I might think it spammy at 100 a day, but 10-20 just feels like an active participant. And it balances out those of us who never submit links, but do comment.
Re: ClickHouse: An open-source column-oriented database management system
#34wish someone would do a comparo between recent versions of CH, Druid and Pinot. (There's a pretty detailed one on the web but it is a couple years old and there's new features in all of them)
Re: ClickHouse: An open-source column-oriented database management system
#35Great software. We are managing terabytes of stocks data and realtime market scanners queries across all market (billions of books and timesales) with hundreds of concurrent requests. We were using kdb before, but clickhouse is more scalable, way cheaper and much more easy to grasp for a newbie.
any info on costs?
Re: ClickHouse: An open-source column-oriented database management system
#36Clickhouse is easily my favorite secret weapon. Lowered our costs by ~50% and reduced data processing times by orders of magnitudes.
compared to what? what's the scale of processing - data size, number of transactions etc...? and how much is the cost?
Re: ClickHouse: An open-source column-oriented database management system
#37People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
Take a look at query engines like Trino (formerly PrestoSQL) [ https://trino.io/ ]. (Disclaimer: I'm a contributor to Trino). I used it at a previous job to combine data from MongoDB, Kafka, S3 and Postgres to great effect. It tries to push-down as many operations as possible to the source too to improve performance. Full ANSI SQL support over multiple number of backends (Kafka, Cassandra, Postgres, ClickHouse, S3 an…
Came here to say the same great things about Trino
Re: ClickHouse: An open-source column-oriented database management system
#38Clickhouse is easily my favorite secret weapon. Lowered our costs by ~50% and reduced data processing times by orders of magnitudes.
compared to what? what's the scale of processing - data size, number of transactions etc...? and how much is the cost?
Data size is in TBs and “transactions” are millions
Old system was a hodgepodge of oracle/mssql/postgres with data engineering glue connecting them.
Re: ClickHouse: An open-source column-oriented database management system
#39Great software. We are managing terabytes of stocks data and realtime market scanners queries across all market (billions of books and timesales) with hundreds of concurrent requests. We were using kdb before, but clickhouse is more scalable, way cheaper and much more easy to grasp for a newbie.
You’re ingesting real time data into Clickhouse?
Actually not, but we are ingesting a whole day of EOD market data (thousands of symbols, millions of timesales, billions of books events) in less than a couple of hours. This is 4x faster than the pace of the realtime market (not counting the extended hours due to the limited volume).
Re: ClickHouse: An open-source column-oriented database management system
#40A really nice database. We use it to aggregate web analytical data in real-time for Pirsch [0] and it performs really well. Writing queries for it is close to relational databases, which made the switch from Postgres easier. User data is still managed in Postgres and losely coupled through an ID field.