Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snorkel AI: Putting Data First in ML Development

snorkel.ai

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Snorkel AI: Putting Data First in ML Development

#21

Post by kmax12 »

mactournier wrote: post #4

They totally ignored the open source package in their site. https://github.com/snorkel-team/snorkel Moreover, based on the commit history in github, and their README.md , my understanding is that they will stop supporting their open source repo. Best of luck for them.

If anyone is looking for an open source library in this space, I work on one called Compose (https://github.com/FeatureLabs/compose).

With compose, a user defines a labeling function, and then compose scans the historical data looking for training examples to train a machine learning model.

The library has evolved as we apply it to more and more real world use cases, but it is based on approach in this paper from 2016: https://dai.lids.mit.edu/wp-content/uploads/2016/08/07796929....

Re: Snorkel AI: Putting Data First in ML Development

#22

Post by polm23 »

tadkar wrote: post #20
sanxiyn wrote: post #14

Earlier quoted context omitted.

My understanding is different. I think you are talking about correlated data source. Snorkel's surprising innovation is that it does NOT overweigh correlated data source.

So, I am specifically talking about the scenario where all your labelling functions are highly correlated and there is little or no ground truth data to come up with empirical weights for each of the labelling functions. An example is the scenario where you have the label functions: x>5, x>4.99, x>5.01 for some feature x. I am really struggling to see how Snorkel can correct for the correlation, especially given the…

The Snorkel paper doesn't cover this in depth, the math is all in this paper:

https://arxiv.org/abs/1703.00854

I can't say I followed all the proofs, but it seems that under certain limited assumptions about labelling functions they prove their generative function can do well.

Reading Snorkel it initially sounded like magic in the bad way, but this does make it clear that if your labelling functions are garbage or have certain kinds of problems there's nothing they can do about it.

Even leaving aside the generative model I think the focus on function-based data bootstrapping is great, which is why I've been following Snorkel's projects for a while.

Re: Snorkel AI: Putting Data First in ML Development

#24

Post by eggie5 »

anyone figure out how to do multi-label classification w/ snorkel? It seems like it's current formulation only supports single-label, ie softmax.

I find in practice, especially w/ user-interaction, system most problems are not single-label, but multi-label. Also, in the single-label setting it's often necessary to define an negative "OTEHR" class which is very difficult to define w/ snorkel in my experience.

Re: Snorkel AI: Putting Data First in ML Development

#26

Post by eggie5 »

king_magic wrote: post #23

Yeaaaah, I’m skeptical. Especially since they seem to dance around what Snorkel actually does at pretty much every opportunity.

Snorkel does weak supervision for you. It takes your unlabeled data and use defined labeling functions (LFs), maps the LFs on the data and then de-correlates relates everything to give you a dataset that you can use for multi-class single-label supervision.

It's very powerful.

Re: Snorkel AI: Putting Data First in ML Development

#27

Post by eggie5 »

mactournier wrote: post #4

They totally ignored the open source package in their site. https://github.com/snorkel-team/snorkel Moreover, based on the commit history in github, and their README.md , my understanding is that they will stop supporting their open source repo. Best of luck for them.

To that end the OS project will remain: https://spectrum.chat/snorkel/general/announcing-snorkel-flo...

Re: Snorkel AI: Putting Data First in ML Development

#28

Post by eggie5 »

nobodywillobsrv wrote: post #17

What does it actually do? I know "label data" but anything can do that. Is it just a pipeline system with some helpers for running a couple of ML related functions? Is it UI based? Where do you run it? I know these areas well and got nearly nothing from reading the splash page on the site.

The former project is a python package that labels your data using a weak supervision technique. It's not just a pipeline, it's a sophisticated algorithm that helps combine multiple competing labeling functions by removing reweighing based on correlations vs a naive majority-voting scheme.

When you look at ML models as commodities and the fact that you spend most of your time getting data, cleaning data or labeling data it leads to what they call Data Programming. I imagine this will be a UI where you can manage your dataset, by monitoring something they call Critical Slices.

Re: Snorkel AI: Putting Data First in ML Development

#29

Post by eggie5 »

ca_parody wrote: post #2

Forgive me - but how does this avoid the chicken&egg problem here. Without digging through the promo copy, why would one programmatically label training data to do ML on if they have such a program to label data...

A human will label data according to hand-rules or heuristics. What's the difference is a program labels data according to hand-rules or heuristics.

The down-stream discriminates model's goal is to generalize via supervision.

Re: Snorkel AI: Putting Data First in ML Development

#30

Post by eggie5 »

nl wrote: post #15

I spent some time trying Snorkel (the open source version) and its predecessor DeepDive. It was extremely complicated to get it to do anything beyond the demos, and I was never successful to get it to do anything useful. I ended up implementing some of the ideas myself, but I can't say I had any great success.

That's too bad. Myself and colleagues have had good success using the current snorkel package.
Post reply on HN