Does OLAP Need an ORM
clickhouse.com
Does OLAP Need an ORM
1–10 of 52 posts
Re: Does OLAP Need an ORM
#2> If you’ve got your OLAP schemas as objects in your application code
I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing some sql.
Re: Does OLAP Need an ORM
#3The way my company uses Clickhouse is basically that we have one giant flat table, and have written our own abstraction layer on top of it based around "entities" which are functions of data in the underlying table, potentially adding in some window functions or joins. Pretty much every query we write with Clickhouse tacks on a big "Group By All" at the end of it, because we are always trying to squash down the number of rows and aggregate as aggressively as possible.
I imagine we're not alone in this type of abstraction layer, and some type-safety would be very welcome there. I tried to build our system on top of Kysely (https://kysely.dev/) but the Clickhouse extension was not far along enough to make sense for our use-case. As such, we basically had to build our own parser that compiles down to sql, but there are many type-error edge cases, especially when we're joining in against data from S3 that could be CSV, Parquet, etc.
Side note: One of the things I love most about Clickhouse is how easy it is to combine data from multiple sources other than just the source database at query time. I imagine this makes the problem of building an ORM much harder as well, since you could need to build type-checking / ORM against sql queries to external databases, rather than to the source table itself
Re: Does OLAP Need an ORM
#4> If you’ve got your OLAP schemas as objects in your application code I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing…
I'm one of the Moose maintainers, and yes, most OLAP use cases fall into data warehousing categories where exposing the database to analysts and letting them run loose with SQL is viable. We're seeing more and more that OLAP is becoming a core part of the application stack, for user and agent-facing analytics. There, we see a lot more appetite for building on the analytical stack the way we build on the transactional one.
Re: Does OLAP Need an ORM
#5> If you’ve got your OLAP schemas as objects in your application code I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing…
I agree with that being fastest, but not cheapest.
In my experience these one off reports are very brittle. The app ends up making schema changes that are breaking to these one off reports, and you usually don’t find out until it goes to production.
I’ve dealt with the maintenance nightmare before. At current gig we’re exploring solutions, curious what a robust pipeline looks like in 2025.
The ORM piece is interesting — we use ActiveRecord and Ruby, and accidentally breaking schema changes within app will get caught by the unit test suite. I would love for a way to bring OLAP reports in similarly to test at CI time.
Re: Does OLAP Need an ORM
#6My take is...
No one needs an ORM: https://dev.to/cies/the-case-against-orms-5bh4
The article opens with "ORMs have proven to be useful for many developers" -- I believe the opposite is true.
Re: Does OLAP Need an ORM
#7> If you’ve got your OLAP schemas as objects in your application code I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing…
I agree with that being fastest, but not cheapest. In my experience these one off reports are very brittle. The app ends up making schema changes that are breaking to these one off reports, and you usually don’t find out until it goes to production. I’ve dealt with the maintenance nightmare before. At current gig we’re exploring solutions, curious what a robust pipeline looks like in 2025. The ORM piece is interestin…
I mean, if you're relying on tests to catch schema changes... then test your sql reports? This doesn't seem like an amzingly cool solution but if that's the one you're already using...
Re: Does OLAP Need an ORM
#8My take is... No one needs an ORM: https://dev.to/cies/the-case-against-orms-5bh4 The article opens with "ORMs have proven to be useful for many developers" -- I believe the opposite is true.
I think what I just really want is a language that treats sql as a "first class" component in the same way perl treats regexes.
The devil is of course in the details, but it's a nice dream.
Re: Does OLAP Need an ORM
#9> If you’ve got your OLAP schemas as objects in your application code I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing…
I agree with that being fastest, but not cheapest. In my experience these one off reports are very brittle. The app ends up making schema changes that are breaking to these one off reports, and you usually don’t find out until it goes to production. I’ve dealt with the maintenance nightmare before. At current gig we’re exploring solutions, curious what a robust pipeline looks like in 2025. The ORM piece is interestin…
Why not test the OLAP reports?
Surely there is a way to run a raw query in Rails/ActiveRecord and use it in a smoke test?
Re: Does OLAP Need an ORM
#10> If you’ve got your OLAP schemas as objects in your application code I guess I have a wildly different interpretation of typical OLAP scenarios. To me this acronym mostly means "reporting". And in 99% of cases where the business desires a new report, the ideal views or type systems have not been anticipated. In these cases (most of them), I can't imagine a faster way to give the business an answer than just writing…
Without doubt, the majority of the market for OLAP today is still internal warehousing & BI. But the market for using OLAP behind features inside user-facing B2C/B2B apps has been kicking off for quite a few years now. Big consumer apps like Stripe, Uber, Shopify...pretty much every B2B SaaS with a usage/metrics dashboard...they're usually punting queries off to an OLAP to populate those stats/charts. That's where something like this might come in handy, I can't imagine it being using for general internal reporting (in the current form, anyway.)