Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is BI dead? – On dismantling data's ship of Theseus

benn.substack.com

Post reply on HN
Print view View original

71–80 of 118 posts

Re: Is BI dead? – On dismantling data's ship of Theseus

#71

Post by eyeball »

pram wrote: post #30

I’ve spent the last 10 years of my life building “Big Data” platforms for companies (Kafka, Druid, Hadoop, Databricks, Presto, Teradata, Informatica, MicroStrategy, etc etc) and I still have no idea what people need most of this crap for. There’s so much duplicative functionality, and a basically never ending list of OLAP adjacent products emerging every year. I mean it’s good for me personally, it’s pretty much a gr…

I wonder about this a lot. The IT department where I work is rolling out a new “data platform”. I’m in the finance unit and a prime customer but fairly up to date on the tech. This thing has soooo many logos on it. I suspect there is someone just saying yes to everything their consultant pitches.

Like why have databricks, azure synapse, AND snowflake on the same picture. I’m certain I could do everything they say they’re going to do with this shit with an integration tool, snowflake, a machine to run python on, and our BI tool in the same cloud.

Re: Is BI dead? – On dismantling data's ship of Theseus

#72

Post by dgudkov »

>BI shouldn’t handle data ingestion and storage

I've been working in the BI industry for almost 20 years and I don't know why the author believes that the "Original BI" includes ETL/storage. He mentions BusinessObjects and Microstrategy as an example of traditional BI tools popular 20 years ago, but neither of them had ETL/storage capabilities. Both were basically visual SQL generators. (Although, BusinessObjects later acquired an ETL company). Qlik has a bit of everything - ETL, columnar storage, and dataviz. But that's more of an exception than a rule.

Nevertheless, the author has a point - BI seems to not have a clear path forward. There are experiments with natural language queries (won't work), automatic text summary generation (useless), AI-assisted automatic insights (might be a nice feature, but barely a "product"), and a few more. The self-service story of BI never really took off (I believe self-service ETL is a more interesting story [1]). Neither did story-telling. Dashboards are oversold and are mainly used to impress CxOs to close big deals. Geospatial dataviz is useful, but has limited application.

I would envision two possible directions to advance for the BI industry:

1) Analytical notebooks, something like Jupiter Notebooks but no-code and for BI might be a good idea to explore, but I haven't seen anything like that.

2) No-code/lo-code analytical app building. Reporting and dashboarding tools are just highly specialized app builders. Why not make a step further and generalize it a bit?

[1] https://easymorph.com (Disclaimer: it's my company)

Re: Is BI dead? – On dismantling data's ship of Theseus

#73

Post by akdor1154 »

eyeball wrote: post #71
pram wrote: post #30

I’ve spent the last 10 years of my life building “Big Data” platforms for companies (Kafka, Druid, Hadoop, Databricks, Presto, Teradata, Informatica, MicroStrategy, etc etc) and I still have no idea what people need most of this crap for. There’s so much duplicative functionality, and a basically never ending list of OLAP adjacent products emerging every year. I mean it’s good for me personally, it’s pretty much a gr…

I wonder about this a lot. The IT department where I work is rolling out a new “data platform”. I’m in the finance unit and a prime customer but fairly up to date on the tech. This thing has soooo many logos on it. I suspect there is someone just saying yes to everything their consultant pitches. Like why have databricks, azure synapse, AND snowflake on the same picture. I’m certain I could do everything they say the…

> Like why have databricks, azure synapse, AND snowflake on the same picture.

This hits painfully close to home. Our org just "finished" such a build, mainly done by a large contractor, and the result is a messy nightmare.

> integration tool, snowflake, a machine to run python on, and our BI tool

Yep, my next campaign will be to gradually pare it down to this. Snowflake is wonderfully capable.

Re: Is BI dead? – On dismantling data's ship of Theseus

#74

Post by edmundsauto »

perl4ever wrote: post #58
FranzFerdiNaN wrote: post #42

Earlier quoted context omitted.

I work as a data engineer in a BI team, with our product being part of a large software suite in healthcare. We got a small number power users who really use the dashboards, who get real result from them and who actively help our development. We also got a larger group of people who use it once in a while. But most people seem to log in once or twice and then never again. And if you ask why it’s often because they co…

>they continue to work on their own personal homegrown BI-suite in excel. Why aren't you providing your dashboards via Excel, then? What exactly does the fancy BI software do that Excel won't? I have used Qlikview but not Tableau or Informatica, and I'm a bit fuzzy on what people mean when they say "Power BI". Is using Power Query what you would call Power BI? Or does that mean using the data model stuff? Power Query…

PowerBI is kind of a MS Access for BI. It loads data from your warehouse and gives both visual builders and query interfaces to data. IE, it's a specific product name - the data viz tool from MS to accompany their other services.

Re: Is BI dead? – On dismantling data's ship of Theseus

#75

Post by easton »

perl4ever wrote: post #58
FranzFerdiNaN wrote: post #42

Earlier quoted context omitted.

I work as a data engineer in a BI team, with our product being part of a large software suite in healthcare. We got a small number power users who really use the dashboards, who get real result from them and who actively help our development. We also got a larger group of people who use it once in a while. But most people seem to log in once or twice and then never again. And if you ask why it’s often because they co…

>they continue to work on their own personal homegrown BI-suite in excel. Why aren't you providing your dashboards via Excel, then? What exactly does the fancy BI software do that Excel won't? I have used Qlikview but not Tableau or Informatica, and I'm a bit fuzzy on what people mean when they say "Power BI". Is using Power Query what you would call Power BI? Or does that mean using the data model stuff? Power Query…

Power BI is a tool from Microsoft, mostly unrelated to Power Query (although you can pull data from Power BI into Excel via Power Query, but you can also do that with just about anything tabular).

https://powerbi.microsoft.com/en-us/

Re: Is BI dead? – On dismantling data's ship of Theseus

#76

Post by mberning »

jl6 wrote: post #31

The ugly truth of BI tools is that they are of most benefit to the organizations which are least capable of using them. The goal of properly exploiting BI comes with many prerequisites which sound superficially reasonable but turn out to be decade-long side-quests. Things like having all your data in one data model. Things like understanding where your data comes from, and exactly what it means. These prerequisites a…

This is so true. I worked a BI tooling startup and it was a lot of fun. But the big clients all had the same types of issues which ultimately dooms their quest. At some point the CFO, CTO, CEO notices they are spending more on “BI” than marketing and they start to get very curious about what they are getting for their money. It’s never quite what they expect.

Re: Is BI dead? – On dismantling data's ship of Theseus

#77

Post by the_af »

pram wrote: post #35
the_af wrote: post #32

Earlier quoted context omitted.

Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a m…

It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)

Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?

Re: Is BI dead? – On dismantling data's ship of Theseus

#78

Post by perl4ever »

easton wrote: post #75
perl4ever wrote: post #58

Earlier quoted context omitted.

>they continue to work on their own personal homegrown BI-suite in excel. Why aren't you providing your dashboards via Excel, then? What exactly does the fancy BI software do that Excel won't? I have used Qlikview but not Tableau or Informatica, and I'm a bit fuzzy on what people mean when they say "Power BI". Is using Power Query what you would call Power BI? Or does that mean using the data model stuff? Power Query…

Power BI is a tool from Microsoft, mostly unrelated to Power Query (although you can pull data from Power BI into Excel via Power Query, but you can also do that with just about anything tabular). https://powerbi.microsoft.com/en-us/

"The first release of Power BI was based on the Microsoft Excel-based add-ins: Power Query, Power Pivot and Power View" (https://en.wikipedia.org/wiki/Microsoft_Power_BI)

Power Query is the only part of this that has seemed useful to me.

The Data Model and DAX in particular seemed to me like distractions that I wasted time on before discovering the M language.

Saying Power Query is mostly unrelated demonstrates that the confusion is not all mine.

Re: Is BI dead? – On dismantling data's ship of Theseus

#79

Post by the_af »

geodel wrote: post #41
pram wrote: post #35

Earlier quoted context omitted.

It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)

Indeed. The idea that somehow Kafka is beyond reproach seems amusing to me.

Thankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.

Re: Is BI dead? – On dismantling data's ship of Theseus

#80

Post by dcsan »

blakeburch wrote: post #68
monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

Wonderful comment. But trying to automate actions (write/modify underlying data or system) is often a hugely complex task compared to a visualization that allows you to form an insight. Have you found enough common/repeatable examples that you can automate or develop some type of actions template? What is your business domain?
Post reply on HN