I have been doing this for my company, when there is a sector you have data on - there is also external factors like how much of the market does your db cover for prices etc. and NLP on different items and metrics for different types of data in your offerings. What do you do with this when the actual field names are very general (i.e. item metric region unit)
Launch HN: Syndetic (YC W20) – Software for explaining datasets
31–36 of 36 posts
Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets
#32I have been doing this for my company, when there is a sector you have data on - there is also external factors like how much of the market does your db cover for prices etc. and NLP on different items and metrics for different types of data in your offerings. What do you do with this when the actual field names are very general (i.e. item metric region unit)
We actually built in the concept of "display names" for exactly this purpose. You want to keep the actual field name in the schema so that ingestion works properly, but you also want to describe the field as helpfully as possible. In another comment, Steve mentioned that we are trying to tackle the first problem you mention (how much overlap is there between my dataset and other external factors) with the concept of creating dataset intersection or relevance scores.
Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets
#33This reminds me of readme generator for Frictionless 'Data Package' https://frictionlessdata.io/
Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets
#34This reminds me of readme generator for Frictionless 'Data Package' https://frictionlessdata.io/
Interesting, I just came across Frictionless recently as part of the FISD group that's trying to implement a set of standards across the financial industry for documenting datasets.
Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets
#35Makes a ton of sense to me, I really wish the datasets that I work with had this or some kind of equivalent.
Would be really great if you could generate a fake dataset of similar size for testing purposes. It would take some thinking but it would be really useful for building a consumer before getting the full dataset.
Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets
#36Makes a ton of sense to me, I really wish the datasets that I work with had this or some kind of equivalent. Would be really great if you could generate a fake dataset of similar size for testing purposes. It would take some thinking but it would be really useful for building a consumer before getting the full dataset.
I missed this comment earlier - we can pilot with sample/fake data to give the user a sense of how the system works and what workflows within it work best for their company. The versioning/diffs wouldn't make much sense, but that's probably ok.