I hoped it was a follow up on yesterday's comment on this. Thanks for sharing, Mr misterdata!
I made advanced BI queries with Scratch puzzle pieces
11–20 of 21 posts
Re: I made advanced BI queries with Scratch puzzle pieces
#12Few years ago I essentially went the same way though only up to a demo (because someone else got the contract) , but the target was to modify the ETL pipeline that before was implemented in hard coded T-SQL stored procedures and a Java app whose source code was evidently lost and recreated with disassembler.
The constructs in Blockly would have been used to generate the transform queries that turned input data into all kinds of summaries in snowflake schema
Re: I made advanced BI queries with Scratch puzzle pieces
#13I'd like to take a moment to appreciate that the first thing I saw on this page was a GIF explaining exactly what the blog post was talking about, but visually. I didn't even have to scroll down, and 60 seconds later, I had a much better idea of what this post was communicating.
Re: I made advanced BI queries with Scratch puzzle pieces
#14I'm surprised there's no mention of Ab Initio, which looks like [1], in these threads. AFAIK they were the pioneer in BI ETL while everyone else was copying.
Then again, they are pretty secretive, and that may be why I can't find any videos of the tool itself in use (edit here's one [3]), maybe due to copyright takedown requests.
That software was the successor of Thinking Machines [2], which was the hot AI company of the 80s AI boom. The software itself is quite good at parallelizing logic. And, the graphical front-end makes it easy for non-programmers to pick up the tool.
[1] https://3.bp.blogspot.com/_FwFkbVFfnGQ/S1qa8lgcw4I/AAAAAAAAA...
[2] https://en.wikipedia.org/wiki/Thinking_Machines_Corporation
Re: I made advanced BI queries with Scratch puzzle pieces
#15Probably a bit of a tangent, but the BI world sure loves their no-code tools. It's one of the few sub-industries where they really took hold.
I do a lot of BI and am a coder. I find these tools (Tableau) super frustrating. So much point and click. Such poor abstractions. Not at all DRY. I like the end product just not the process. Lacks a good API for doing it programmatically - which makes sense as they make money selling desktops.
The reason for point and click is that you need to be able to experiment and iterate quickly, which is what point and click is good for (ie tweaking knobs). BI is about asking lots of questions of your data and seeing which ones provide useful results.
If you already know what question you have in mind, then yeah, it's going to be a bit tedious.
Re: I made advanced BI queries with Scratch puzzle pieces
#16Earlier quoted context omitted.
I do a lot of BI and am a coder. I find these tools (Tableau) super frustrating. So much point and click. Such poor abstractions. Not at all DRY. I like the end product just not the process. Lacks a good API for doing it programmatically - which makes sense as they make money selling desktops.
The reason for point and click is that you need to be able to experiment and iterate quickly, which is what point and click is good for (ie tweaking knobs). BI is about asking lots of questions of your data and seeing which ones provide useful results. If you already know what question you have in mind, then yeah, it's going to be a bit tedious.
Exactly, mainly because nobody has a clue of their own data, so fast iteration is a must, and engineering in the IT departments supporting these solutions are some of the most clueless engineers I met.
Re: I made advanced BI queries with Scratch puzzle pieces
#17Probably a bit of a tangent, but the BI world sure loves their no-code tools. It's one of the few sub-industries where they really took hold.
I do a lot of BI and am a coder. I find these tools (Tableau) super frustrating. So much point and click. Such poor abstractions. Not at all DRY. I like the end product just not the process. Lacks a good API for doing it programmatically - which makes sense as they make money selling desktops.
Well in good tools (Tableau being one of them) there is always the raw SQL query somewhere (ideally with parameters) so that you can do the heavy lifting in code but style use the GUI for quick and nice interactive rendering.
PS: Well some tools are even cooler like QGis for map making. The GUI of this FOSS tool is heavily parametrizable and fully extensible/scriptable in Python/QT.
There is quite not not yet an equivalent in BI world...
Re: I made advanced BI queries with Scratch puzzle pieces
#18Earlier quoted context omitted.
I mostly agree, but I think there's a reason behind the madness. Back in the early 00s to early 10s they really were more productive, "secret alien technology"-type tools. But these days obviously ten lines of python (or whatever else) calling the database do exactly the same thing, except you actually have git, debuggers, ides, etc. Many BI departments still cling to them because they're comparing 2020s no-code tool…
Tableau is way faster to demonstrate your work on. These professions need to get shit done fast. The workplaces that need them are extremely reactive to the market, etc. So having debuggers isn’t necessarily a thing they’re interested in. Now if you’re in a slower paced BI environment that’s where you generally see more traditional programming tooling be used.
It surely is fast enough, I'm not blaming people who use Tableau.
But "faster" compared to what? It's not faster than pandas/seaborn, not faster than d3... even just for super simple stuff where GUI tools shine.
If the final solution needs to be somewhat self-service, then sure, you can't expect the end user to write python or javascript, and Tableau is fine. If the end result is just a report that some technician has to prepare, then I sincerely doubt it's the fastest way to get there, even ignoring maintainability, source control, etc, where it's just a no contest.
Re: I made advanced BI queries with Scratch puzzle pieces
#19Old iron, 666 ~
Re: I made advanced BI queries with Scratch puzzle pieces
#20Probably a bit of a tangent, but the BI world sure loves their no-code tools. It's one of the few sub-industries where they really took hold.
Not because they're any better of a fit for BI stuff mind you. At my place of work, the BI department is a black hole you can keep shoving more compute into, and they'll just come up with worse queries.
It's still early in the space, but there are real advantages to defining visualisations, metrics, and reports in a code-first way. Aside from being able to do more powerful things with something like pandas as opposed to Tableau, being able to check your dashboards into git, create reusable libraries internally, review changes in PR format etc., goes a long way to making things reproducible and less of a black hole. Also, creating reports and dashboards programmatically opens up some interesting use-cases where a drag and drop tool would break down.
We're building an open-source framework for creating reports and dashboards using Python, which you might find helpful: https://github.com/datapane/datapane. You can think of Datapane as the view layer / interface for any BI analysis you're doing using the open-source Python ecosystem. Any feedback would be much appreciated!