Uses and abuses of cloud data warehouses

Why does bolding words imply adtech?

Real-time analytics for human is not that useful. Human can't make decisions and take actions in minutes anyway, let alone seconds. A notable exception can be log analytics for operations, but I'd argue in that case throughput is more more important than a few seconds of latency. Case in point, CloudWatch Insights can consistently drive about 1GB/s of log scan. It's good enough for log search in practice.

On the other hand, real-time analytics for machines can be critical to a business, which is why Yandex built Clickhouse and ByteDance deployed more than 20K nodes of Clickhouse.

Just like using any technology, we need to figure out what problems we solve for real-time analytics first.

Well, it's an engineering decision, so there is no direct answer.

But as an engineering manager, at least I must ask and answer the following question: even when nobody wants _slow_ data, how fast is fast enough? I don't see decision makers choosing and thinking better with a 10 min latency vs 20 min latency, as they are not looking at the reports all the time, even for big events like Black Friday (they have meetings and stuff you know, even their supporting analyst teams do).

For more time-critical matters (i.e. real time BI or real time automatic microdecision making for fraud detection), as I said, we did have the capability to run both more frequent microbatches or do RT processing using Flink connected directly to our app backend messaging system (ironically Confluent, Kafka as a Service). But that is very different to using a complex real time log as Kafka running on "pet" servers as the cornerstone of your data platform and then propagating said data to different engines/datastores (at least 4 as I said) for downstream processing. That's a lot of moving parts running in a low reliability environment.

Overengineering is a thing, and I think it was my responsability at the time to limit the level of complexity considering the reality of the business and the resources we had in the team, even if that meant 20 minutes of latency for a business report. That's my point and why I say I think it was a bad decision to use a Kafka based stack. YMMV obviously.

> Operational workloads have fundamental requirements that are diametrically opposite from the requirements for analytical systems, and we’re finding that a tool designed for the latter doesn’t always solve for the former.

We aren't even going to consider the other direction? Running your analytics on top of a basic-ass SQL database?

In our shop, we aren't going for a separation between operational and analytical. The scale of our business and the technology available means we can use one big database for everything [0]. The only remaining challenge is to construct the schema such that consumers of the data are made aware of the rates of change and freshness of the rows (load interval, load date, etc).

If someone wants to join operational with analytical, I think they shouldn't have to reach for a weird abstraction. Just write SQL like you always would and be aware that certain data sources might change faster than others.

Sticking everything onto one target might sound like a risky thing, but I find many of these other "best practices" DW architectures to be far less palatable (aka sketchier) than one big box. Disaster recovery of 100% of our data is handled with replication of a single transaction log and is easy to validate.

[0]: https://learn.microsoft.com/en-us/azure/azure-sql/database/h...

Do they seriously charge in the millions for a server to run a database? A single box? Fucking oracle man.