do you mind if I derail the thread for a little bit?
[if yes]
I currently work at a small-medium sized company, and we have one BI guy. Our ops team is ~5 people, and 1-2 of the more junior guys are tasked with doing everything necessary to support the BI guy. which basically means making sure the BI software is installed properly on the container, configured properly, has access to the db, and it comes up properly etc
BI is a critical business function, so it's a pretty high priority, but the process is way less automated and skillfully crafted than the other parts of the system, mainly cause everyone hates it and only barely understands what the BI system (in this case, pentaho) actually does. it's no coincidence that the most junior engineers often end up doing it (with occasional but pretty infrequent help from the seniors when blocked)
what i've been wondering is, what exactly does "a BI guy" need their system to do? I read a bit about ETL systems, but haven't done much research. I assume the job of an ETL system is to pull data from the production database(s), clean it up as necessary, and use that data to calculate whatever business metrics are required? is the software more about making it easy for the BI team to use an interface to craft their queries/dataflow/etc?
basically my motivation is I've mulled over the idea of having the ETL system be written in-house. the benefits being potentially smoother and more native operation and in a way that's more seamless to integrate with monitoring, alerting, automated jobs etc. my guess is that such a project would end up being a waste of time in 98% of cases, especially because you need a usable frontend for the BI guy.
so, if I may, what exactly is business intelligence / what do you do on a day-to-day basis? I'm pretty experienced with traditional finance/investing jobs, eg reading 10-q's, 10-k's, balance sheets etc, and using the data to calculate cash flow, opex, gross margin, and every other metric you could conceivably want. But I'd never heard of "business intelligence" until starting an internship several months ago.
thanks, and no worries if you don't have time to respond / if this is too off-topic.