Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is BI dead? – On dismantling data's ship of Theseus

benn.substack.com

Post reply on HN
Print view View original

81–90 of 118 posts

Re: Is BI dead? – On dismantling data's ship of Theseus

#81

Post by perl4ever »

edmundsauto wrote: post #74
perl4ever wrote: post #58

Earlier quoted context omitted.

>they continue to work on their own personal homegrown BI-suite in excel. Why aren't you providing your dashboards via Excel, then? What exactly does the fancy BI software do that Excel won't? I have used Qlikview but not Tableau or Informatica, and I'm a bit fuzzy on what people mean when they say "Power BI". Is using Power Query what you would call Power BI? Or does that mean using the data model stuff? Power Query…

PowerBI is kind of a MS Access for BI. It loads data from your warehouse and gives both visual builders and query interfaces to data. IE, it's a specific product name - the data viz tool from MS to accompany their other services.

It seems to be part of Excel since 2016, no? It appears to me that it's morphed over time and I'm not sure to what extent it remains a separate product.

I've used both DAX and M (Power Query) via recent versions of Excel. Obviously there are plenty of ways to interface with data.

https://en.wikipedia.org/wiki/Power_Pivot says both Power Query and Power Pivot are part of Excel.

Re: Is BI dead? – On dismantling data's ship of Theseus

#82

Post by the_af »

geodel wrote: post #40
the_af wrote: post #32

Earlier quoted context omitted.

Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a m…

Well ideas behind all the other stuff will also make lot of sense if one align with perspective of snake oil salesman. And I do think Kafka/hadoop etc fit in snake oil category. Linkedin saying Kafka is good for them is not saying much.

Did you read the article? Because it makes a ton of sense regardless of whether it's LinkedIn saying whatever. It reads more like a paper than an advertisement -- which is more than what can be said about many articles peddling "solutions".

Re: Is BI dead? – On dismantling data's ship of Theseus

#83

Post by dikei »

haddr wrote: post #27
marcinzm wrote: post #25

Earlier quoted context omitted.

I run a data team. We do two things: * Provide a clean minimal view of the data that other teams can pivot table on top of or download in the BI tool. We do onboarding, support and all that for this data. * Provide dashboards and reports for more complicated ongoing questions that require data that's not cleaned yet. We don't use Looker but based on the sales demo it seems optimized for this type of workflow. The cor…

This is the direction where most BI work is heading for: you need a team of data engineers that can make sure that the data for your BI dashboards is delivered and in a good quality. Everything else that doesn’t need it is probably simple enough to be run as some PowerBI dashboard on top of production DB. I think today most effort in those heavier BI cases is spent on data transformations and making sure that data fl…

If you're not careful, the data engineers team is often overloaded with requests from the analysis team.

With big data warehouse, more often than not, the fancy dashboards would not run fast enough without a dedicate pipeline that materialize data before hand. Too many useless dashboards and your data team would not have enough manpower to do anything else other than operating the existing pipelines. As a result, new dashboards become slower than old dashboards due to lack of maintenance/optimization, and everyone complain about the poor performance.

Re: Is BI dead? – On dismantling data's ship of Theseus

#84

Post by pram »

the_af wrote: post #77
pram wrote: post #35

Earlier quoted context omitted.

It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)

Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?

There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire)

Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.

Re: Is BI dead? – On dismantling data's ship of Theseus

#85

Post by gitgud »

monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

Amplitude has been incredible, at my new workplace. Fast and lots of pretty graphs are easily generated. Also used by tonnes of different people throughout the company!

Re: Is BI dead? – On dismantling data's ship of Theseus

#86

Post by blakeburch »

dcsan wrote: post #80
blakeburch wrote: post #68

Earlier quoted context omitted.

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

Wonderful comment. But trying to automate actions (write/modify underlying data or system) is often a hugely complex task compared to a visualization that allows you to form an insight. Have you found enough common/repeatable examples that you can automate or develop some type of actions template? What is your business domain?

Previous domain I worked in was digital advertising. We built solutions to automate ad and keyword creation from product feeds and promo calendars, increase/decrease bids and budgets based on hourly data, verify all settings were set to best practices, etc. on Google/Bing/Facebook. We built internal systems that allowed us to map any data view in our warehouse to any API endpoint in a templated fashion.

The technology is getting good enough with cloud warehouses that most of the logic can be defined in SQL, with a script only needing to map the results back to an API.

Current domain is SaaS. I've found many of the same types of opportunities on the sales/marketing side of things, but not at quite the same scale.

Agreed that automation it will always take more time than a visualization. If we can shift the conversation towards the expected action that will be taken off the dashboard, we can hopefully save thousands of wasted hours on visualizations that don't get used, redirecting those hours towards more impactful efforts.

I would love to expand my mindset and hear of domains where the complexity is far higher. Feel free to shoot me an email.

Re: Is BI dead? – On dismantling data's ship of Theseus

#87

Post by jimmySixDOF »

monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

>perhaps AR/VR will provide a novel and more valuable approach

Yes. A lot of darts are being thrown as to where XR fits on which Hypecycles, but one of the most promising uses to me is as a new enduser engagement layer for BI(g) data. The new UI/UX possibilities of immersive 3D have a low level effect on what/how information can be rendered in meaningful ways. It's still early days in this space, but here is a quick list of some work in progress for anyone interested:

[1] https://flowimmersive.com/ Focused on Data driven story-telling through an inhouse 3D visualization tool. Strong on AR collaboration and responsible for "The Data Guy" popular on TikTok.

[2] https://www.badvr.com/ Industry focused on using XR for specific applications like 5G Radio Coverage heatmap modeling and Smart City interfacing tools.

[3] https://3data.io/ Use of XR for IT Operations Center applications (NOC/SOC)

[4] https://github.com/ACEMS/r2vr This project lets you output basic WebXR 3D visualizations using R

Re: Is BI dead? – On dismantling data's ship of Theseus

#88

Post by viraptor »

blakeburch wrote: post #68
monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

The data driven process app sounds similar to the ideas behind Knime (https://www.knime.com/) to me.

I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.

Re: Is BI dead? – On dismantling data's ship of Theseus

#89

Post by esyir »

monkeydust wrote: post #17

As someone who runs teams deploying BI to internal stakeholders (product, sales) I am pretty fed up with them. Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer. We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data mun…

On the flipside though, I've had some success with self service bi platforms like metabase.

Allowing stakeholders some leeway to conduct their own independent analysis (after a short training session) has allowed our strained data team to hand over simpler analysis to focus on the harder problems.

Re: Is BI dead? – On dismantling data's ship of Theseus

#90

Post by goodlinks »

I appologise in advance for probably using poor terminology..

Maybe I am a naive fool but IMHO the benifits of BI are reconciliation of the business model:

- reconciling processes - reconciling data - reconciling governance

Its a painful process to impliment and you will hit all of your compliance, governance and change management issues head on.

IMHO if you accept that this is what you are doing then its potentially very valuable for your business, but it must be done humbly with humilty as every single measure and many dimensions are each a project on their own.

The pretty graphs are largely irrelevant apart from making employees feel valued and to re-enforce abstractions via visual metaphors.

I get pissed off when end users are not trained in consuming the database warehouse through tools like excel or access. This project should include general IT education programme for most companies.

Also.. if the end users cannot use a pivot table then you need to teach them that and prototype a bunch of stuff in excel before you go anywhere near BI of any flavour.

Post reply on HN