Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is BI dead? – On dismantling data's ship of Theseus

benn.substack.com

Post reply on HN
Print view View original

101–110 of 118 posts

Re: Is BI dead? – On dismantling data's ship of Theseus

#101

Post by the_af »

pram wrote: post #84
the_af wrote: post #77

Earlier quoted context omitted.

Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?

There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazi…

I agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!

Re: Is BI dead? – On dismantling data's ship of Theseus

#102

Post by marcinzm »

dspillett wrote: post #100
haddr wrote: post #27

Earlier quoted context omitted.

This is the direction where most BI work is heading for: you need a team of data engineers that can make sure that the data for your BI dashboards is delivered and in a good quality. Everything else that doesn’t need it is probably simple enough to be run as some PowerBI dashboard on top of production DB. I think today most effort in those heavier BI cases is spent on data transformations and making sure that data fl…

> dashboard on top of production DB A lot of what users need can be done this way in my experience. Also giving their people access to customise reports/exports can go even further. I'd advise against direct access to “real” production though: have the dashboard and custom reports run against a (probably readonly) replica of some sort¹ so runaway reports don't impact live app performance. [1] via replication, log shi…

>A lot of what users need can be done this way in my experience.

In my experience, no, as the company gets to any complexity and size. The production DB is often optimized for use by the ORM/language of choice or there's twenty of them for microservices. This makes it really awkward to do some things in SQL. So now your engineering team is getting requests to keep changing the prod DB schema so analytics can be done more easily. More over the prod DB is likely 10% of the data your business needs. There's a reason companies like Fivetran exist to slurp fifty data sources into a warehouse. Marketing wants google and facebook and mailchimp data. Operations wants saleforce data. Finance wants stripe data. And they want all of this tied to the inhouse data.

Sure you can make reports in each of those services but then you need to tie them together and then people complain of differences in metrics and then you have a bad time.

edit: Not to mention analytics on what users do. Always fun when there's a billion row table in your production db for that.

Re: Is BI dead? – On dismantling data's ship of Theseus

#103

Post by vgeek »

viraptor wrote: post #88
blakeburch wrote: post #68

Earlier quoted context omitted.

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

The data driven process app sounds similar to the ideas behind Knime ( https://www.knime.com/ ) to me. I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.

This. Knime is similar to BI tools, but it is more action oriented. The reports may not be as pretty, but it can handle the entire ETL pipeline and actually do things besides make interactive pivot tables.

Re: Is BI dead? – On dismantling data's ship of Theseus

#104

Post by Scalestein »

monkeydust wrote: post #96
SilverRed wrote: post #63

Earlier quoted context omitted.

We provide power bi access to the data our app runs on and it has been a mixed bag. The customers absolutely love it. But they build dashboards where hitting the refresh button can bring a super expensive db to its knees for half an hour. So then we have to get our db people in to rewrite all of their reports so they don't do this which eliminated the benefit of having the customers do it. In the end there is just no…

OP here. Interesting as we're looking at going long with PowerBI embedded for a client facing app and ditching our home grown effort which is always playing catch-up and costs a fortune.

I did the same thing last year for a client. So far it has worked pretty well. Client analysts create the reports and then deploy them for end users to view through a browser. End users can't do toooo much regarding playing with the data. It is slightly better than static data but the real value has been the client analysts being able to modify it without needing dev help (no JS chart libraries to mess with!)

Re: Is BI dead? – On dismantling data's ship of Theseus

#105

Post by blakeburch »

guareber8 wrote: post #98
blakeburch wrote: post #68

Earlier quoted context omitted.

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

The way I'm (currently!) approaching this is by implementing the decision engine in 2 steps separated by an evaluation period. I refer to these to the stakeholders as "user approves" and "user reviews". Essentially, I first implement the engine with the output being an automated list of 1-click actions (typically a set of links on an email, but could easily be a dashboard button or anything else), so the system defau…

I think that's a fantastic approach. You're effectively avoiding the pitfall of automating too soon while still shifting the focus towards "driving action with this data". If people click the buttons frequently, it's ripe for full automation - and they already trust it. If not, there's still an easy way to for users to get the job done quicker.

Is this all internally built?

Re: Is BI dead? – On dismantling data's ship of Theseus

#106

Post by blakeburch »

viraptor wrote: post #88
blakeburch wrote: post #68

Earlier quoted context omitted.

I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data. When it comes to business stakeholders, the biggest obstacle is trust. If you're n…

The data driven process app sounds similar to the ideas behind Knime ( https://www.knime.com/ ) to me. I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.

While I'm not super familiar with Knime, it looks like it falls into the legacy BI category that Benn talks about. An "all-in-one" for your data.

Is the platform still easily used by business users? Or does it primarily become a gated way for Data Engineers to build/action on data?

A lot of decisions can't be easily visualized in your standard dashboard. You can forecast the potential of those automated decisions, but that's still black box logic. You really need more of a middle ground that lets you preview "given these inputs, show me what the output would be".

Re: Is BI dead? – On dismantling data's ship of Theseus

#107

Post by intrasight »

jon-wood wrote: post #52
intrasight wrote: post #10

I remember learing decades ago that "In UNIX, everything is a file" I approach BI now as "In BI, everything is in your database". With everything in the database (SQL Server in my case) I can deliver BI to my users with quality tools like Tableau and Power BI - or just with generic web reports.

It really is amazing how far you can get by just dumping everything into a single database and letting people do joins. We’re doing the same thing with Stitch replicating data from production databases and a few SaaS products into a big Postgres instance and it’s so low on maintenance hassle. The most advanced our “BI pipeline” gets is a few stored procedures run from a cron job to aggregate data in a particularly la…

Exactly. I do BI at a Fortune 500. We have most everything anyone would care about in a database. Our job is to load and curate, create views and write queries.

Re: Is BI dead? – On dismantling data's ship of Theseus

#108

Post by zhendlin »

A lot of conventional BI was focused on 'single player' mode -- basically an analyst running analysis at their computer and maybe dropping that graph into a presentation. This fundamentally limited the usefulness of of BI results (they got stuck in time and rarely updated in) and made follow-ups hard since the graph was no longer connected to the data source.

All the while things like Google Docs and Slack made collaboration around documents and ideas much easier with @mentions, threads, etc.

So BI can be a lot more useful if it is 1\ accessible anywhere (not just desktop or a crappy mobile app) 2\ collaborative -- bringing the mentions, shared questions, and ability to make decisions together 3\ not crazy expensive (looking at your tableau + looker)...because at these high price points the tool ends up not getting offered to everybody, it ends up being more limited in who gets access and hence less useful across a company

Full disclosure, I built Zing Data which is an app for mobile first business intelligence and is free for small teams. Works with PostgreSQL, MySQL, and Google BigQuery. https://www.getzingdata.com/

Would love any feedback folks have -- we're actively improving it and I'm sure this community has a lot of great ideas!

Re: Is BI dead? – On dismantling data's ship of Theseus

#109

Post by perl4ever »

Rastonbury wrote: post #91
perl4ever wrote: post #81

Earlier quoted context omitted.

It seems to be part of Excel since 2016, no? It appears to me that it's morphed over time and I'm not sure to what extent it remains a separate product. I've used both DAX and M (Power Query) via recent versions of Excel. Obviously there are plenty of ways to interface with data. https://en.wikipedia.org/wiki/Power_Pivot says both Power Query and Power Pivot are part of Excel.

It's a separate product, analogous/competitor to Tableau. Just go to the product page

I assume you're referring to https://powerbi.microsoft.com/ ?

What I'm seeing there looks as though it is targeted at people who make the decision to buy products like this, but not to use them. I'm not seeing what couldn't be done with Excel 365 alone. There are screenshots with verbiage about how well it works with Excel. And they look like...Excel.

I guess it supports R, and maybe makes it easier to seamlessly refresh a report. That's aggravating though, that it makes me suspect Excel might be purposely rough around the edges in order to not displace Power BI.

As I've said a few times, I like M and wish they'd commit to it.

I honestly suspect Microsoft is repackaging functionality that already existed for many years in their "ball of mud" software, with a shiny new wrapper and better defaults, but practically unusable, because people who license software are easily manipulated.

Power Automate is the most awful example of this I've used so far.

You say it's a competitor to Tableau, well, Tableau is a competitor to Qlikview, and I've used Qlikview and Excel enough to consider Excel better in every way that I care about. (My organization uses SharePoint, so that's how reports are usually distributed)

Re: Is BI dead? – On dismantling data's ship of Theseus

#110

Post by viraptor »

blakeburch wrote: post #106
viraptor wrote: post #88

Earlier quoted context omitted.

The data driven process app sounds similar to the ideas behind Knime ( https://www.knime.com/ ) to me. I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.

While I'm not super familiar with Knime, it looks like it falls into the legacy BI category that Benn talks about. An "all-in-one" for your data. Is the platform still easily used by business users? Or does it primarily become a gated way for Data Engineers to build/action on data? A lot of decisions can't be easily visualized in your standard dashboard. You can forecast the potential of those automated decisions, bu…

> Is the platform still easily used by business users?

I don't know and honestly I don't believe there's a system that actually achieves it. But this it does go beyond BI in actually being aimed towards running code based on results. Which makes it closer to Labview-with-graphs than to BI.

Post reply on HN