Live data from Hacker News

Snorkel AI: Putting Data First in ML Development

snorkel.ai

61–67 of 67 posts

Re: Snorkel AI: Putting Data First in ML Development

#61
post #56

Earlier quoted context omitted.

At the ODSC presentation I went to last year where the team presented they actually used a vision problem as their canonical example. It's hard to grasp without a concrete example, but it makes a lot of sense the way they explained it. For example, lets assume you want to identify something like a lung tumor. So you have many MRI images and they're all largely the same template of image. Using traditional image proce…

Yeah, but for some tasks, like recognizing a car, or something else you'd have to write pretty sophisticated code to even get to a reasonable result. It's far easier in those cases to use supervised learning to have a NN learn to do it for you.

Agreed! As noted in other answer, Snorkel certainly does not work for everything :) And indeed, in many cases it may be easier to express what you know extensionally (label examples) vs intensionally (write functions). A lot comes down to the unit cost per label over time- and whether it's more economical to label a bunch of data by hand vs. write LFs or similar.

That's btw why a lot of examples of ML today are ones where data is (i) simple for non-experts to label, (ii) non-private and therefore easy to outsource for labeling, and (iii) low rate of change (e.g. images for self-driving, basic NLP stuff for chat bots, etc)- this kind of data can be labeled cheaply and once, so hand-labeled training sets are (barely) economically feasible to build manually. However, most data is not that easy or cheap to label, needs to be relabeled constantly to adapt to change, and thus the investment in a programmatic approach is often far better even if certainly not push-button!

Re: Snorkel AI: Putting Data First in ML Development

#62
post #52

Earlier quoted context omitted.

Lurking for a few more min... great question! Dealing with both class imbalance and issues of pernicious biases in both underlying data distributions and training labels is an extremely important topic. Our underlying theory deals with local biases (e.g. individual labeling functions or sources of training signal being biased) but systemic biases (e.g. the user driving the system being biased) are certainly tougher.…

Thanks for the answer! The local vs systemic bias thing is particularly interesting for a system like snorkel. I have a clarifying question. I'm imagining a dumb example like recipes, where "1 tsp salt" is a common format for ingredient. I'd imagine that the majority of ingredients follow that format, so it'd be a natural function to write. I'd also imagine that there's a correlation between following that format and…

Great question! Let me rephrase so you can confirm my understanding: I have some labeling functions (LFs) that are far more accurate on a majority subset of the data than on one or more minority subsets or "slices" of the data... and these subsets are not necessarily correlated with the class labels, so this isn't a traditional class imbalance problem...

We've actually done some recent work on this (https://papers.nips.cc/paper/9137-slice-based-learning-a-pro...) where we have users define these critical "slices" approximately so that the model being trained can pay special attention to them (extra representation layers) so they don't get drowned out by the majority subsets/slices. But definitely a lot more to do in this area!

Re: Snorkel AI: Putting Data First in ML Development

#63
post #52

Earlier quoted context omitted.

Thanks for the answer! The local vs systemic bias thing is particularly interesting for a system like snorkel. I have a clarifying question. I'm imagining a dumb example like recipes, where "1 tsp salt" is a common format for ingredient. I'd imagine that the majority of ingredients follow that format, so it'd be a natural function to write. I'd also imagine that there's a correlation between following that format and…

Great question! Let me rephrase so you can confirm my understanding: I have some labeling functions (LFs) that are far more accurate on a majority subset of the data than on one or more minority subsets or "slices" of the data... and these subsets are not necessarily correlated with the class labels, so this isn't a traditional class imbalance problem... We've actually done some recent work on this ( https://papers.n…

Cool idea, and thanks for the answer! I'll have to look more closely at the paper :)

Re: Snorkel AI: Putting Data First in ML Development

#65
post #53

Earlier quoted context omitted.

yeah, you can do single-label w/ snorkel, but not multi-label. Multi-label snorkel would be the killer feature bc making the negatives (ie for a softmax) is very hard especially when you work w/ user-interaction systems with an unknown negative distribution.

You can always do multi-label as a multi-task learning model (or just a set of binary models), which is something we (and many others) have explored before! A lot of the adjustments for mainline Snorkel have to do with (A) the semantics of the labeling functions (need to be able to express that something is not class A and/or have a general per-class prior) and (B) all the infra to support what is just now a bunch of…

Snorkel has a label mutual exclusion assumption right?

My core problem is a multi-label problem, but my snorkel data, from the LabelModel is inherently single-label (mutually exclusive). What is the prevailing recommendation to do multi-label w/ Snorkel? Is the below what you are currently recommending?

For a given, k-wise multi-label problem:

1. Generate k binary datasets w/ LabelModel 2. Train k separate binary classifiers for each respective dataset 3. At inference/prediction time pass input though the k classifiers and get scores.

Is this what the current recommendation is? Create a set of binary classifiers?

Re: Snorkel AI: Putting Data First in ML Development

#66
post #38

I presume this does not apply to computer vision datasets? Frankly I am still confused at what exactly Snorkel does.

you have a dataset of images and you write code (labeling functions LF) to label the images. Snorkel handles the pipeline but more importantly corrects the conflicts/correlations between the LFs. The output is a supervised dataset w/ mutually exclusive labels a la softmax classification.

the labels are noisy, but you have a quantity that you could not get by humans, AND at a faster/cheaper rate. they provide analysis arguing that, for discriminative models, quantity CAN outweigh quality.

to your point it's not typically used w/ the image-only modality. It's mostly used where there is some meta-data attached.

Re: Snorkel AI: Putting Data First in ML Development

#67
post #54

Earlier quoted context omitted.

Thanks Alex. I'm sure you can relate that when you have a an unbounded input distribution (like w/ user-interaction systems), defining that other class w/ current snorkel is difficult/impossible.

Yeah definitely- and would love to chat sometime, as this is a space I've at least had less direct hands-on interaction with. There's a line of work in the ML literature on "Positive unlabeled (PU) learning"--basically, setting where there are only positive labels or abstains--with a lot of theoretical ties to what our stuff rests on, I think a tie in here is interesting. Of course, most of these approaches rely on s…

Thanks for the lead on PU Learning.

I signed up for a demo of the new platform, looking forward to chatting. Me and a colleague from work spoke w/ Henry last year about a potential partnership but I guess it got lost in the mix...

Post reply on HN