Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Syndetic (YC W20) – Software for explaining datasets

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#21

Post by stevepike »

mason55 wrote: post #18

Any plans on a hosted version? Either regular on prem or as something that can be privately hosted on e.g. AWS like DataBricks or Snowflake? I love the idea but we could never expose most of our data to a public SaaS. There are all kinds of restrictions we have on things like data privacy and data needing to stay in specific regions.

Yeah, on-premise or at least private cloud has come up a few times. Beyond the data privacy and licensing requirements it'd also just be plain faster in some cases. We haven't offered it yet just because we're a small company and are rapidly adding features. Our backend is mostly running in k8s so I don't think it'll be a _huge_ technical rewrite to get it running in private cloud. Frankly I just don't have experience supporting software running outside my control and want to make sure we take the time to do it correctly.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#23

Post by SamuelAdams »

I wonder if you could combine your service with freely available datasets like Google Dataset Search [1] to demo what a large amount of various datasets would look like under your service.

[1]: https://datasetsearch.research.google.com/search?query=puppi...

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#24

Post by aswihart »

SamuelAdams wrote: post #23

I wonder if you could combine your service with freely available datasets like Google Dataset Search [1] to demo what a large amount of various datasets would look like under your service. [1]: https://datasetsearch.research.google.com/search?query=puppi...

We would love to do that at some point. There is tons of open data out there, but not a lot of it has useful descriptions at the field level (only the dataset level), so it would take some time to put a collection together that is robust. Also the demo on our splash page is a demo of the artifact we create (i.e. the published dictionary) only. The other side of the web app is a management layer to bundle datasets into collections, annotate fields, configure sample sets, and share the artifacts. We'll work on fleshing out our demo to show what the system looks like when there are hundreds or thousands of datasets.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#25

Post by mason55 »

stevepike wrote: post #21
mason55 wrote: post #18

Any plans on a hosted version? Either regular on prem or as something that can be privately hosted on e.g. AWS like DataBricks or Snowflake? I love the idea but we could never expose most of our data to a public SaaS. There are all kinds of restrictions we have on things like data privacy and data needing to stay in specific regions.

Yeah, on-premise or at least private cloud has come up a few times. Beyond the data privacy and licensing requirements it'd also just be plain faster in some cases. We haven't offered it yet just because we're a small company and are rapidly adding features. Our backend is mostly running in k8s so I don't think it'll be a _huge_ technical rewrite to get it running in private cloud. Frankly I just don't have experienc…

Cool. We're an enterprise B2B SaaS company that does a ton of data interchange with our customers. Each project burns a bunch of engineering hours and calendar time because customers send us data that doesn't match the spec or we just don't have anyone outside the engineering org who can confirm that the data looks how it's supposed to.

Something which simplified the process of analyzing sample data and provided a view of it to a non-technical user would be very valuable. But as I mentioned in my previous comment we could never expose any of our data outside of our private infrastructure, so we can't use this until there are other options for hosting.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#26

Post by stevepike »

pplonski86 wrote: post #19

What data types do you plan to support? Have you considered supporting datasets with images+labels used in computer vision? How would you like to handle them? Are you going to support data labeling tasks?

We've talked with some image data providers who are creating datasets for use in machine learning. We're not running any of our own models on image data right now, so I think the place we can be most useful is in summarizing metadata about the images in cases where the dataset isn't just an image file. For example, if the dataset is images of intersections plus bounding box coordinates of street signs we can tell a prospective consumer what % of images have a street sign in them. If you have a little more metadata (e.g., what time of day the photo is) the stats get much more useful out of the box.

I don't understand the data labeling question. How would you imagine us getting involved there?

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#27

Post by stevepike »

nocitrek wrote: post #17

Does this support more complex data structures - for example parquet files?

The manual data upload is restricted to well-formed CSV files w/ headers on the first row right now. For the "contact us" higher tier we'll handle any file format that we can extract columns from, so parquet would be fine.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#28

Post by carterehsmith »

stevepike wrote: post #12
dodata wrote: post #9

Neat! Congrats on the launch - the demo is very helpful to understand the product. Having consumed long, painful PDF data dictionaries in the past, this is a big breath of fresh air. Excited to see where Syndetic goes! For me, the most painful part of working with 3rd party data was actually figuring out the "match rate" to internal data. For example, you might be a consumer-facing company who hopes to add more conte…

Yes! This has come up across multiple industries and is probably the feature on our roadmap I'm most excited about. The implementation is tricky but customers definitely care about the intersection of a provider's data with their own. Some more sophisticated providers have internal tools for generating things like sample sets customized to a prospect. We're going to be adding a feature where we can flag fields as ide…

That sounds very useful.

I am also super impressed that you managed to present your product without mentioning "big data" or "machine learning" or AI - given that anyone that does anything these days crams those big words in.

Thats is good, good luck.

Re: Launch HN: Syndetic (YC W20) – Software for explaining datasets

#29

Post by stevepike »

carterehsmith wrote: post #28
stevepike wrote: post #12

Earlier quoted context omitted.

Yes! This has come up across multiple industries and is probably the feature on our roadmap I'm most excited about. The implementation is tricky but customers definitely care about the intersection of a provider's data with their own. Some more sophisticated providers have internal tools for generating things like sample sets customized to a prospect. We're going to be adding a feature where we can flag fields as ide…

That sounds very useful. I am also super impressed that you managed to present your product without mentioning "big data" or "machine learning" or AI - given that anyone that does anything these days crams those big words in. Thats is good, good luck.

Thanks, I'm with you on the big data buzzwords and trying to avoid overengineering things (one of my favorite HN posts ever is https://news.ycombinator.com/item?id=8908462).

Right now the data scanning is just a fork of https://github.com/BurntSushi/xsv/ running in a container with plenty of ram, and we've handled files in the ~20GB range with no problem. I think we could actually scale up to ~100GB files with xsv, which seems to cover 99%+ of data providers we're running in to. Providers might be processing massive amounts of data but the eventual deliverable they share with their customers is rarely too big for one machine.

That said, we will probably move away from our super simple stack towards running a Spark cluster in the medium term. Not for "big data" (actually I expect Spark to have higher latency and possibly to be slower for a moderate sized dataset than the rust solution) but because we want to be able to run multiple parallel scans over the datasets for upcoming future features. Some of that will involve a DAG of dependencies (e.g., do type detection first to figure out fields that are categorical, then generate visualizations where the plots are grouped by whatever the values are of the categorical field). There are also a bunch of nice libraries in the spark world for more comprehensive stats.

Post reply on HN