Data Catalogs Are Dead; Long Live Data Discovery
montecarlodata.com
Data Catalogs Are Dead; Long Live Data Discovery
1–8 of 8 posts
Re: Data Catalogs Are Dead; Long Live Data Discovery
#2Some points are good but I challenge the assumption that our data is a mess because of “unstructured data”. In fact the one example the author makes is of highly structured data - salesforce. The issue I see on my end is that we keep buying solutions focusing mostly on functionality and not evaluating how such solutions store and manage the data they produce. Lack of integrated solutions results in complicated data pipelines due to having to jump through a lot of hoops to blend thing together.
Re: Data Catalogs Are Dead; Long Live Data Discovery
#3I look after a lot of data. We have gone thru many different ideas and solutions. Data catalog is the only working solution we found after we POC ton of vendor solutions.
I'm not AI scientist, but features generated by models are not structured?
Re: Data Catalogs Are Dead; Long Live Data Discovery
#4I look after a lot of data. We have gone thru many different ideas and solutions. Data catalog is the only working solution we found after we POC ton of vendor solutions. I'm not AI scientist, but features generated by models are not structured?
Perhaps this will shed some light: https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over...
Re: Data Catalogs Are Dead; Long Live Data Discovery
#5Anyone have a good data lineage story while still using mainstream data processing and ETL tools like Spark/Flink/etc.? I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque.
To me, that's the core part (As the article also mentions) that allows data discovery. Otherwise you're just guessing with heuristics [0]. It also allows someone to go from map [Input Table(s) => This Table's Fields] which I think is half the battle in understanding a table.
[0] Not to say guessing is the worst, just that the information is already in the ETL code so why are we guessing?
Re: Data Catalogs Are Dead; Long Live Data Discovery
#6Anyone have a good data lineage story while still using mainstream data processing and ETL tools like Spark/Flink/etc.? I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque. To me, that's the core part (As the article also mentions) that allows data discovery. Otherwise you're just guessing with heuristics [0]. It also allows someone to go from map [In…
Have you heard of Monte Carlo? Data lineage + data catalog + data anomaly detection: https://www.montecarlodata.com/product/
Re: Data Catalogs Are Dead; Long Live Data Discovery
#7Anyone have a good data lineage story while still using mainstream data processing and ETL tools like Spark/Flink/etc.? I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque. To me, that's the core part (As the article also mentions) that allows data discovery. Otherwise you're just guessing with heuristics [0]. It also allows someone to go from map [In…
Have you heard of Monte Carlo? Data lineage + data catalog + data anomaly detection: https://www.montecarlodata.com/product/
Yeah, that's who the post was written by (Which I actually did read).
> I assume they may be selling something like that, but their "integration with my existing data stack" is pretty opaque.
The issue is that I have no clue what that actually looks like in practice and I'm not going to request a random demo to find out.
Re: Data Catalogs Are Dead; Long Live Data Discovery
#8I gather that data catalogs, a rectangle on the diagram, is dead and data discovery, an ellipse on the diagram takes over. Is there anything more to this article than geometric shape of buzzword containers?