61–70 of 196 posts
I would assume there's an automated test suite, but also some way of diffing large amounts of input data and visualizing those input additions relative to model classifications?
What are the common tools for this?
Earlier quoted context omitted.
What is your take on the need for time-travel queries for versioned, mutable data? Versioning immutable data items is not enough if you have structured data that is updated. Every time you update a data item, you store a full copy - not a diff of the actual data. You are not able to make "time-travel queries" - give me the data that was generated in this time-range, for example. For example, if you have a Feature Sto…
The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.
There's also litetree, whose slogan is simply "SQLite with branches":
Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…
Earlier quoted context omitted.
The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.
That's nice. Do you have any idea if it is possible to translate those rows into higher-level time-travel queries? Like if you could plugin an adapter to transform the rows into a data structure (parquet, arrow, json, whatever) that could be useful to analytics and ML apps?
AS OF: https://www.dolthub.com/blog/2020-03-20-querying-historical-...
HISTORY SYSTEM TABLE: https://www.dolthub.com/blog/2020-01-23-access-to-everything...
You can run `dolt q -r csv -q ` to output whatever you want to a CSV. We would need to do work to output a hierarchical format.
I'm sure it's possible to build whatever time travel operation you want. We can produce an audit log of every cell in the database pretty quickly.
Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…
This is Git for data. It's a program, and it appears to be an open-source one you can download and use today.
GIN: https://gin.g-node.org/ datalad: https://www.datalad.org/
At the time GIN looked really promising as something potentially simple enough for end users in the lab but with a lot of power behind it. (Unfortunately we never got it deployed due to organizational constraints... but that's a separate story.)