Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is BI dead? – On dismantling data's ship of Theseus

benn.substack.com

Post reply on HN
Print view View original

111–118 of 118 posts

Re: Is BI dead? – On dismantling data's ship of Theseus

#111

Post by mcrad »

blakeburch wrote: post #68
monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

I've found that the vast majority of stakeholders want a dashboard either to (1) spend other people's money and feel important or (2) create opportunities for multiple "version of the truth" and thus prepare multiple narratives to pick and personally gain from when needed. Middle managers would rather reserve the right to say X dashboard has a metric that doesn't reconcile with Y other reliable source, than to actually collaborate with competent analysts and build a holistic info platform. We need to stop casting such a wide net and selling the dream of a glamorous all-knowing BI tool and instead, service analytics on a case by case basis. Over time this can inform practical dashboard design, kept up by a small team of experts and largely ignoring the "laundry lists" of every prospective user.

Re: Is BI dead? – On dismantling data's ship of Theseus

#112

Post by CRConrad »

dgudkov wrote: post #72

>BI shouldn’t handle data ingestion and storage I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage. He mentions BusinessObjects and Microstrategy as an example of traditional BI tools popular 20 years ago, but neither of them had ETL/storage capabilities. Both were basically visual SQL generators. (Although, BusinessObjects lat…

> I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage.

I've done some BI, but mostly DW/ETL, for over twenty years... And I can't remember ever seeing ETL viewed as a sub-category of BI in any project I've been part of. It hasn't even been mentioned as such -- except in weird articles like this -- this side of the turn of the century. Before that, yes, the categories were muddled. But not since then.

> 2) No-code/lo-code analytical app building. Reporting and dashboarding tools are just highly specialized app builders. Why not make a step further and generalize it a bit?

Sure. I've long thought the best tool, one that could do both ETL and BI -- is an IDE with a full-blown programming language and a good library: Something like D̵e̵l̵p̵h̵i̵ Free Pascal / Lazarus.

Re: Is BI dead? – On dismantling data's ship of Theseus

#113

Post by CRConrad »

blakeburch wrote: post #69

The interesting point here is that the unbundling of BI is resulting in tools for every occasion. What I'm seeing on my end is that the "modern data stack" is starting to become very fragmented as a result. Sure, each of these tools are better suited to accomplishing a very specific task with your data. But the problem is that now these tools aren't talking to each other. No one has a single place to view their data…

Don't worry, you won't have to invent a whole new thing: There is a whole sub-industry of IT called Data Warehousing that does exactly this.

Re: Is BI dead? – On dismantling data's ship of Theseus

#114

Post by CRConrad »

intrasight wrote: post #107
jon-wood wrote: post #52

Earlier quoted context omitted.

It really is amazing how far you can get by just dumping everything into a single database and letting people do joins. We’re doing the same thing with Stitch replicating data from production databases and a few SaaS products into a big Postgres instance and it’s so low on maintenance hassle. The most advanced our “BI pipeline” gets is a few stored procedures run from a cron job to aggregate data in a particularly la…

Exactly. I do BI at a Fortune 500. We have most everything anyone would care about in a database. Our job is to load and curate, create views and write queries.

Did you just invent Data Warehousing? If so, I have bad news for you...

Re: Is BI dead? – On dismantling data's ship of Theseus

#115

Post by intrasight »

CRConrad wrote: post #114
intrasight wrote: post #107

Earlier quoted context omitted.

Exactly. I do BI at a Fortune 500. We have most everything anyone would care about in a database. Our job is to load and curate, create views and write queries.

Did you just invent Data Warehousing? If so, I have bad news for you...

I do know that most project fail, and I suspect that a big part of that is having people focus on shiny new tools instead of just using a database and hiring some developers.

Re: Is BI dead? – On dismantling data's ship of Theseus

#116

Post by CRConrad »

intrasight wrote: post #115
CRConrad wrote: post #114

Earlier quoted context omitted.

Did you just invent Data Warehousing? If so, I have bad news for you...

I do know that most project fail, and I suspect that a big part of that is having people focus on shiny new tools instead of just using a database and hiring some developers.

Yeah, I was mainly just being facetious about how what you described has "been a thing" for decades, and there's a whole industry around it (in which I've earned my living this century). So if you thought your insight was new, you'd be disappointed... :-)

But, come to look at it in hindsight, my comment may have been misplaced -- maybe it should have gone as a reply to the same comment you replied to, with its "It really is amazing how far you can get..." At least if they meant "amazing" in the sense of "surprising". To me, what's amazing and surprising is only how many people don't already know this.

Re: Is BI dead? – On dismantling data's ship of Theseus

#117

Post by agallego »

pram wrote: post #84
the_af wrote: post #77

Earlier quoted context omitted.

Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?

There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazi…

for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda.

    https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets

Re: Is BI dead? – On dismantling data's ship of Theseus

#118

Post by infinite8s »

dgudkov wrote: post #72

>BI shouldn’t handle data ingestion and storage I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage. He mentions BusinessObjects and Microstrategy as an example of traditional BI tools popular 20 years ago, but neither of them had ETL/storage capabilities. Both were basically visual SQL generators. (Although, BusinessObjects lat…

> 2) No-code/lo-code analytical app building. Reporting and dashboarding tools are just highly specialized app builders. Why not make a step further and generalize it a bit?

Yes exactly! Dashboarding tools only get you halfway - there's no easy way to take action from within the dashboard itself. I see analytic app building as a cross between Tableau and Airplane/Retool

Post reply on HN