See also http://www.transformy.io/#/app Ok, just realized somehow the site has vanished. Not working archived version: http://web.archive.org/web/20161028231256/https://www.transf...
https://apps.synesty.com/transformy/en-us this might be it
Transform Data by Example [video]
41–50 of 94 posts
Re: Transform Data by Example [video]
#42I'm playing around with a client-side js implementation of this at https://www.robosheets.com/ It's not production ready / launched yet, but it's getting there. I'd be interested to finds (or really doesn't find) this useful :)
I wasn't able to figure out how to use the app, but I did want to drop in to say that I like how you've placed the "Upgrade to enable" notices in cells past a limited range.
Re: Transform Data by Example [video]
#43You know what this reminds me of? Those trained neural-net things which, however many training examples you give it, always seem to find some way to “cheat” and not do what you want while still obeying all your training data correctly. Something like this: Suppose we have a table of strings of digits, some including spaces, and we’d like to remove the spaces. From 123 456 234567 345 678 to 123456 234567 345678 Now, w…
Sounds like you're talking about over-fitting the data. Or perhaps just not providing an evaluation function that is sufficiently general. All ML can fall into this trap.
Re: Transform Data by Example [video]
#44You know what this reminds me of? Those trained neural-net things which, however many training examples you give it, always seem to find some way to “cheat” and not do what you want while still obeying all your training data correctly. Something like this: Suppose we have a table of strings of digits, some including spaces, and we’d like to remove the spaces. From 123 456 234567 345 678 to 123456 234567 345678 Now, w…
This is why I want any ML device to be able to explain itself. It could train on your before-and-after examples and come up with a list of what it thinks you want it to do. For your example, it could list: “Remove interior spaces from each item” or it could say: “Remove the middle character from any 7-character strings to make them 6 characters in length” You would be able to do something with that.
> This is why I want any ML device to be able to explain itself.
This is the problem of lacking explanatory mechanisms in ML.
Note that some techniques that are very out of vogue at the moment, such as Genetic Programming, are much better than neural nets in this regard.
Re: Transform Data by Example [video]
#45Earlier quoted context omitted.
Neural nets are infamous for not doing this. Learning algorithms that produce decision trees are usually used in this situation.
This might be a dumb question, but let's say that for whatever reason on a specific problem it's much easier to train a neural network that generalizes well than a decision tree. Why not train the network, then build an equivalent decision tree that just tries to reproduce the network's output? When building the tree from the network, overfitting would not be a concern. In fact, you'd want it to overfit. You could ev…
> This might be a dumb question, but let's say that for whatever reason on a specific problem it's much easier to train a neural network that generalizes well than a decision tree. Why not train the network, then build an equivalent decision tree that just tries to reproduce the network's output? When building the tree from the network, overfitting would not be a concern. In fact, you'd want it to overfit.
You haven't fixed anything here. You've just encoded your training data in a neural net and then presented the same problem to the decision tree learner. Unless you're planning to transform your training data somehow?
Re: Transform Data by Example [video]
#46Earlier quoted context omitted.
Neural nets are infamous for not doing this. Learning algorithms that produce decision trees are usually used in this situation.
This might be a dumb question, but let's say that for whatever reason on a specific problem it's much easier to train a neural network that generalizes well than a decision tree. Why not train the network, then build an equivalent decision tree that just tries to reproduce the network's output? When building the tree from the network, overfitting would not be a concern. In fact, you'd want it to overfit. You could ev…
[deleted]
Re: Transform Data by Example [video]
#47I would love something similar for Google Spreadsheet.
Re: Transform Data by Example [video]
#48Earlier quoted context omitted.
Neural nets are infamous for not doing this. Learning algorithms that produce decision trees are usually used in this situation.
It's theoretically possible for a neural net to do this; the network just needs to have the explanation as an output. I agree that decision trees would be more reliable and easier to train, but I'm not sure if hardcoding every feature is scalable.
> the network just needs to have the explanation as an output
And how would you evaluate whether the explanation was correct or not?
Re: Transform Data by Example [video]
#49You know what this reminds me of? Those trained neural-net things which, however many training examples you give it, always seem to find some way to “cheat” and not do what you want while still obeying all your training data correctly. Something like this: Suppose we have a table of strings of digits, some including spaces, and we’d like to remove the spaces. From 123 456 234567 345 678 to 123456 234567 345678 Now, w…
MS had been experimenting with this for a while[1]. They even included this in Excel 2013 as "FlashFill". It does not use any NN/ML at all. It uses "program synthesis", which by definition can tell you exactly what "program" it has synthesized to convert you data. In fact in you example it would not cheat, rather leave the string unchanged as explained in the paper. [1] https://www.microsoft.com/en-us/research/public…
More generally, anything by Sumit Gulwani's group at MSR should be of interest.
Re: Transform Data by Example [video]
#50You know what this reminds me of? Those trained neural-net things which, however many training examples you give it, always seem to find some way to “cheat” and not do what you want while still obeying all your training data correctly. Something like this: Suppose we have a table of strings of digits, some including spaces, and we’d like to remove the spaces. From 123 456 234567 345 678 to 123456 234567 345678 Now, w…
I agree, thank you for peovoking this thought. It is raw and if so I apologize.
This is where hinting is important. Metadata. That sequence if I know it's a phone number, or a sequence of increasing digits, depends a lot on metadata.
Given some reasonable sample size, i believe machine learning could provide hints as to some of the common types of formats. Semi automated data hinting or structuring?
There is a bidirectional connection between interpreting your data and how your data is structured
Is it possible to use your data column to statistically hint at metadata characteristics by some sort of clustering, then use that to automatically clean input data?