11–20 of 21 posts
here is the extension if anyone is interested https://github.com/G-Research/spark-extension/blob/master/DI...
Another way of doing this that I've been wanting to implement for a while is to implement the DIFF operator in Apache Calcite[0]. Using Calcite, DIFF could be implemented as rewrite rules to generate the appropriate SQL to be directly executed against the database or the DIFF operator can be implemented outside of the database (which the original paper shows is more efficient).
I am working on this for a different definition of term dataset. I started learning deep learning which led me to start building datasets. Wanting to store versions of the datasets efficiently I started building a version control system for them. It tracks objects and annotations and can roll back to any point in time. It helps answer questions like what has changed since the last release and which user made which ch…
Have you looked into existing version control systems for data, such as DVC?
First the good. Git LFS solves the issue of checking out a massive repository in whole.
Git can work pretty well if your annotations are in a text based format and stored one annotation per file. That makes it easy to track and attribute annotation changes.
What I'm building can serve as a backend to labeling. There is a built in workflow for reviewing changes, objects have different statuses (in annotation, included in release, etc.), reproducible releases, things like that.
It is really designed for collaboration with untrusted third parties. Imagine someone making a pull request for a binary annotation format. To review it you would have to clone it, load it in an annotation tool, then go and tie what you saw to what is in the pull request. What do you do it like 90% of the annotations are correct? Reject everything? Very tough, also assumes your annotater can make a pull request.
Mine will still require you to bring your own annotation tool, but makes it much easier to integrate the review process.
Things like: what Lambdas existed in a customer AWS account 6 months ago in us-east-2 that had access to a specific SQS queue" (because we learned later that one of the consumers of that queue would actually consume Python pickles if you asked nicely, and hence get you RCE).
As a side effect, we do a lot of data diffing: just mostly on more vanilla Clojure structures rather than data sets in the Datasette/CSV/... sense.
For example, we have tooling (e.g. [recidiffist], which we also have wired up to Terraform + S3, so if you write some files to S3, you can get the structured diffs right next to it for free). It's one of those things that's simple and works ridiculously well. Well, if you do it consistently anyway. "Let's look at what resources aren't managed by IAC, and how that has changed historically" is an interesting question to me, but I couldn't do it without having that data available, encoded in a way that didn't pre-suppose how it would be used.
Even though we mean "explanation" differently, this really is critical for enabling explanation and exploration. While many of these pieces of data (e.g. "historical complete snapshots of AWS environments") aren't big compared to a lot of corporate data lakes or log management systems, they're way bigger than what fits in my tiny human brain at once. They're still amenable to fast local analysis. We use [clerk] for amazing notebooks to aid in that.
Where this all comes together is in the encoded, and machine-evaluatable, expressions. The way you do a Cloud audit is you write a Clojure program: you develop invariants about the environment that are or at least should be true, and then you talk about the places that didn't work out.
[recidiffist]: https://github.com/latacora/recidiffist [clerk]: https://github.com/nextjournal/clerk
One use case I've not heard anyone talk about is modeling/optimization. In my previous role we were managing a bunch of data that all fit nicely in databases (e.g. market prices, timeseries data from facilities, optimization model structure) but had no good tools to summarize "what changed from last week" when the optimization models would spit out something unusual.
This looks very interesting. Having not read the paper or the author’s library, I am curious how well it scales to more features. I would naively assume it is doing an exponential number of column comparisons. Keeping in mind the strong correlation between divorces and margarine ( https://tylervigen.com/spurious-correlations ) I am still tempted to wire this up to automatically generate reports when data falls outsid…
I read the paper. It’s fundamentally I think an exponential problem but they apply some constraints to reduce it: only considering tuples of size n Personally I think the case where you’re doing this on one column is not that interesting: for each column, get all values with >= support frequency, group by it on old and new, include in results is risk_ratio over threshold. List results in order of risk_ratio. Going fr…
Earlier quoted context omitted.
Have you looked into existing version control systems for data, such as DVC?
Thanks for the suggestion. I have glanced through the docs in past but haven't tried it. I am trying to do a bit more than what git can offer. First the good. Git LFS solves the issue of checking out a massive repository in whole. Git can work pretty well if your annotations are in a text based format and stored one annotation per file. That makes it easy to track and attribute annotation changes. What I'm building c…
(full disclosure: I'm one of the creators)
Dolt is working in this space, and I think there is a lot of potential: https://www.dolthub.com/ One use case I've not heard anyone talk about is modeling/optimization. In my previous role we were managing a bunch of data that all fit nicely in databases (e.g. market prices, timeseries data from facilities, optimization model structure) but had no good tools to summarize "what changed from last week" when the optimiz…
Dolt is working in this space, and I think there is a lot of potential: https://www.dolthub.com/ One use case I've not heard anyone talk about is modeling/optimization. In my previous role we were managing a bunch of data that all fit nicely in databases (e.g. market prices, timeseries data from facilities, optimization model structure) but had no good tools to summarize "what changed from last week" when the optimiz…
Dolt is git for MySql.
TerminusDB is git for RDF/graphs.