I am currently talking to departments to understand the landscape by asking them a few key questions: 1) What data sets have they used in the past year 2) Where and how do they source it (census website, internal databases etc) 3) How frequently do they refresh it and reuse it.
I am starting to see a pattern where some datasets are sourced from internal systems with their own backing databases. It's a bit of a clutter but I can bring these different datasets to a centralized warehouse/datalake like environment that has discovery and querying capability.
However, there is another consistent thread expressed by departmental analysts - "I get this other data from another agency via an excel spreadsheet as an email attachment" which I am struggling to understand what to do with. At this point I am not seeking a tool or a technical solution but rather some meta thoughts from HN tribe members.
Is there merit in trying to centralize/bring together artifacts like spreadsheets together in a centralized place? Has someone done this/tried this before and saw any real benefit from it or is the dynamic/adhoc nature of this type of information exchange better left alone?