Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Syndetic (YC W20) – Software for explaining datasets

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#11

Post by stevepike »

coolsank wrote: post #8

very cool product! I've worked on much smaller datasets with pandas and their inbuilt profiling report method can slow things down to a crawl!. Hoping to see more from you guys :)

A very reductionist version of our company that Allison hates when I use is "csvstat but on the internet" :-). I think the problem of auto-summarizing datasets has hit kind of a local maximum in what pandas dataframe summaries (csvstat is a similar python tool) can do on one machine. We will be able to add much fancier things like sophisticated type classification (e.g., is this field a stock ticker) without burning your CPU.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#12

Post by stevepike »

dodata wrote: post #9

Neat! Congrats on the launch - the demo is very helpful to understand the product. Having consumed long, painful PDF data dictionaries in the past, this is a big breath of fresh air. Excited to see where Syndetic goes! For me, the most painful part of working with 3rd party data was actually figuring out the "match rate" to internal data. For example, you might be a consumer-facing company who hopes to add more conte…

Yes! This has come up across multiple industries and is probably the feature on our roadmap I'm most excited about. The implementation is tricky but customers definitely care about the intersection of a provider's data with their own. Some more sophisticated providers have internal tools for generating things like sample sets customized to a prospect.

We're going to be adding a feature where we can flag fields as identifying keys and index them. We'll start with a simple intersection count ("upload 100 stock tickers, see how many records match"). Then we'll add an interactive feature to let a prospective customer generate all of the stats in the dictionary scoped down to the subset of data they care about. It's important to be able to answer questions like "for the 100 tickers I care about, how many NULLs are there for this other column?".

Maybe someday we'll even get into the more general record linkage problem when there's no reliable matching key.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#13

Post by knes »

Super great idea. I've been talking to data scientist/eng in B2B SaaS space on how we should bring best practices like that to the Sales/Marketing/Business ops world too.

What would you say are the differences between syndetic and qri.io (not affiliated in anyway)

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#15

Post by aswihart »

gkoberger wrote: post #14

Hey! I’m Greg of ReadMe... think Syndetic but for APIs rather than datasets! Congrats on the launch :) I’ll find you at Alumni Demo Day, or feel free to reach out if I can help with anything! And welcome to the war on PDFs :)

This made our day : ) Thank you!

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#16

Post by stevepike »

knes wrote: post #13

Super great idea. I've been talking to data scientist/eng in B2B SaaS space on how we should bring best practices like that to the Sales/Marketing/Business ops world too. What would you say are the differences between syndetic and qri.io (not affiliated in anyway)

Thanks for the kind words! I hadn't seen qri.io before, but from a read of their website I think it's broadly similar to Dolt (https://www.liquidata.co/) which is git for datasets. Kaggle has a similar data hub at https://www.kaggle.com/datasets that's not open source but is in the same space.

Our approach is to scan the datasets wherever they currently live in production rather than being a new way to store the data. The industry seems to have settled on FTP and S3 for now, and we think it's important that we connect to the same exact thing a customer would access. That lets us keep the dictionary up to date automatically without the data providers needing to change their storage infrastructure.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#18

Post by mason55 »

Any plans on a hosted version? Either regular on prem or as something that can be privately hosted on e.g. AWS like DataBricks or Snowflake?

I love the idea but we could never expose most of our data to a public SaaS. There are all kinds of restrictions we have on things like data privacy and data needing to stay in specific regions.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#20

Post by coolsank »

stevepike wrote: post #11
coolsank wrote: post #8

very cool product! I've worked on much smaller datasets with pandas and their inbuilt profiling report method can slow things down to a crawl!. Hoping to see more from you guys :)

A very reductionist version of our company that Allison hates when I use is "csvstat but on the internet" :-). I think the problem of auto-summarizing datasets has hit kind of a local maximum in what pandas dataframe summaries (csvstat is a similar python tool) can do on one machine. We will be able to add much fancier things like sophisticated type classification (e.g., is this field a stock ticker) without burning…

hah! but this is a very interesting area. You're right on the auto-summarizing issue becoming a problem these days with the usage of larger datasets. Data versioning also is starting to become a larger problem and I saw that you guys already have addressed it in your enterprise product. Hoping to see some sort of API-like version for comparison of data troves from different timelines in the future.
Post reply on HN