How to ingest 1B rows/s in ClickHouse

#1

Post by _peregrine_ » Mon, Aug 18, 2025, 3:21 PM UTC

How to ingest 1B rows/s in ClickHouse
tinybird.co

Re: How to ingest 1B rows/s in ClickHouse

#2

Post by trollied » Mon, Aug 18, 2025, 4:07 PM UTC

From yesterday https://news.ycombinator.com/item?id=44933452

Re: How to ingest 1B rows/s in ClickHouse

#3

Post by jandrewrogers » Mon, Aug 18, 2025, 4:12 PM UTC

I don't know about ClickHouse specifically but these days it isn't that hard to ingest >10M rows/sec per server, including parsing, processing, indexing, and storage. You can just throw hardware at it.

Re: How to ingest 1B rows/s in ClickHouse

#4

Post by [deleted] » Mon, Aug 18, 2025, 4:29 PM UTC

[deleted]

Re: How to ingest 1B rows/s in ClickHouse

#5

Post by jauntywundrkind » Mon, Aug 18, 2025, 4:30 PM UTC

Minor quibble but: probably get another 5-15% by upgrading from oldoldstable Debian 11/Bullseye with kernel 5.10 to something modern like Debian 13/Trixie.

Edit: oh these are containers, right; the difference will be far less since the kernel is determined by the host. Still, I find it sad to see old software versions like this used.

Re: How to ingest 1B rows/s in ClickHouse

#6

Post by _peregrine_ » Mon, Aug 18, 2025, 4:43 PM UTC

jauntywundrkind wrote: post #5

Minor quibble but: probably get another 5-15% by upgrading from oldoldstable Debian 11/Bullseye with kernel 5.10 to something modern like Debian 13/Trixie. Edit: oh these are containers, right; the difference will be far less since the kernel is determined by the host. Still, I find it sad to see old software versions like this used.

nice one

Re: How to ingest 1B rows/s in ClickHouse

#7

Post by _peregrine_ » Mon, Aug 18, 2025, 4:43 PM UTC

trollied wrote: post #2

From yesterday https://news.ycombinator.com/item?id=44933452

definitely interesting and related

Re: How to ingest 1B rows/s in ClickHouse

#8

Post by _peregrine_ » Mon, Aug 18, 2025, 4:44 PM UTC

jandrewrogers wrote: post #3

I don't know about ClickHouse specifically but these days it isn't that hard to ingest >10M rows/sec per server, including parsing, processing, indexing, and storage. You can just throw hardware at it.

yeah I mean that's basically what Javi talks about in the post... if you can throw hardware at it you can scale it (ingestion scales linearly with shards)

but the post has some interesting thoughts on how you do the high-scale ingestion while also handling background merge processes, reads, etc.

Re: How to ingest 1B rows/s in ClickHouse

#9

Post by javisantana » Tue, Aug 19, 2025, 5:28 AM UTC

jauntywundrkind wrote: post #5

Minor quibble but: probably get another 5-15% by upgrading from oldoldstable Debian 11/Bullseye with kernel 5.10 to something modern like Debian 13/Trixie. Edit: oh these are containers, right; the difference will be far less since the kernel is determined by the host. Still, I find it sad to see old software versions like this used.

Thanks.

I re-used all scripts and didn't even notice the linux version.

Re: How to ingest 1B rows/s in ClickHouse

#10

Post by logifail » Tue, Aug 19, 2025, 9:13 AM UTC

jauntywundrkind wrote: post #5

Minor quibble but: probably get another 5-15% by upgrading from oldoldstable Debian 11/Bullseye with kernel 5.10 to something modern like Debian 13/Trixie. Edit: oh these are containers, right; the difference will be far less since the kernel is determined by the host. Still, I find it sad to see old software versions like this used.

> probably get another 5-15% by upgrading from oldoldstable Debian 11/Bullseye with kernel 5.10 to something modern like Debian 13/Trixie

For the on-prem/homelab crowd: I upgraded my desktop from Debian 12 to 13 yesterday only to discover my Mellanox ConnectX-3 NIC wasn't supported by the Trixie kernel. From 10GbE to 0GbE in one reboot :/

Have ordered a ConnectX-4 so that should get me going again, of course 100GbE is severe overkill for my needs but since I already have a 100GbE switch I figured why not...