I completely understand the terrors of hidden dependencies on shadow IT. The work I do tends to grow up organically in different areas of every org, so I've hopped between functional business units, centralized business operations, and IT. When working for business units, I'm careful when I introduce technically advanced process dependencies. I either limit it to the level of technical competency that is generally accessible to the team or else I adopt whatever toolset the IT or development side of the house uses. That way there's always the potential for transition and I don't screw my manager/team over if I leave (or get stuck supporting it in perpetuity while I'm there).
This was an example of that. We do client consulting work, but we're only about 2% of the total revenue of our Fortune 500 parent and our IT policies are not conducive to client-driven work requirements. When my manager looked for what we could use when supporting analytics-oriented aspects of client work, tools like Supermetrics[1] were explicitly denied and IT provided virtually no automation/analytics solutions short of a complete Tableau server, which is prohibitively expensive for most of our clients' needs (especially when allocating IT infrastructure & support costs in addition to the license). I'm not a fan of the fragility and monotony of repetitive manual data collection and reporting, nor do clients like to pay for the excessive hours that takes on a recurring basis compared to competing bids (because presumably our competitors can use SaaS tools for automating rote process).
I scrounged around and found a pocket of PowerBI users in our IT's SQL Server team. Connected with the director over that team and got access for myself. Created a proof of concept architecture that provided an agile platform to address the full spectrum of our potential needs from "throw in basic analytics for free and eat the cost" to "maintain a full on data lake getting a deluge of real time data streams". Client users are guests invited to our tenant, log in with their (usually) already-existing Office 365/AzureAD user, can have the $10/user/month PowerBI Pro license added to their account via their corporate IT, our corporate IT, or a credit card expense. All of their ETL and data connectors and credentials and PowerBI models are compartmentalized into their own app workspace and subject to corporate DLP policies and accessible/auditable by Office admins. It's technically unsanctioned/shadow IT, but I ran it by the director of that team in case I needed to tap his team for sporadic support needs in the future, and he was both interested in my proposed usage and perfectly fine giving it his blessing (which is great, because his team manages the permissions I need for provisioning users, adding licenses, and creating workspaces).
The alternative would have been to either lose business due to non-competitive bids, use a combination of unsanctioned SaaS tools that get expensed and subject to disruption when turnover happens and passwords don't get shared or credit card renewals don't occur (fairly common scenario I've inherited), or using development skills to create custom automation that would be unmaintainable when I left.
> Ideally the IT department would take a step back, assess what the users (you the stakeholder) needs are, and provide the solutions. Better Agile.
This works well in some cases, but falls flat on it's face just as often. It's often prohibitively expensive from a budgetary standpoint for a business team to pull in resources for a needs analysis and subsequent project. Business users are far cheaper, so the vast majority of drift between needs and systems capabilities/processes go unresolved because it's cheaper to absorb the inefficiency on the business labor for quite a while before it hits a critical point that justifies the expense of pulling in ops/dev staff. For most business teams, shadow IT is a pragmatic stopgap solution to hit their deliverables, similar to the type of tech debt you see in IT projects that gets kicked down the road until it's suddenly not something that can be kicked anymore. And the process of IT stumbling across a grass roots solution acting as a de facto production dependency is basically a litmus test for when the debt finally comes due and triggers a formal needs analysis and IT project.
As well, that type of needs analysis generally creates localized optimizations or poor user adoption. The current needs will be a reflection of the processes that evolved within the current system and constraints, and addressing the directly will usually just result in a solution that's "a faster horse". Or it goes to the other extreme, where the solution you provide is state of the art and it turns out they have no ability to transition from riding their current horse to piloting the new fighter jet you provided.
A far more effective approach is to creative a collaborative model. Grass roots based solutions exist for a multitude of reasons and won't ever go away. But the concerns of undocumented, unknown, and potentially unmaintained de facto production dependencies existing outside of the purview of IT is a very real concern. So create a migration path. Put in low friction processes, support structures, and policies that give power users on the business side the ability to create sanctioned grass roots solutions. It'll allow business teams to experiment with and build up competencies and informed opinions on newer capabilities and possibilities, allowing for more nuanced and effective needs analysis for major projects. And it also provides enough oversight and management for IT to step in when usage patterns indicate something has become de facto production and warrants a more formal transition to/evaluation by technical and operations stakeholders. And also provides valuable data to ops/IT on where people are needing to create grass root solutions in general, which can better inform their own future systems planning.