Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snorkel AI: Putting Data First in ML Development

snorkel.ai

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Snorkel AI: Putting Data First in ML Development

#51

Post by master_yoda_1 »

sriku wrote: post #13
master_yoda_1 wrote: post #7

Earlier quoted context omitted.

is not this done for years and called synthetic data generation, simulation etc.

Not data generation. Label generation. .. but the charitable interpretation of your question is valid - we've been doing such ensembling to make higher quality models for some time now. It's getting some good structure, practice and tooling around it is what I feel.

Yeh then advertise it as a tool rather than AI. The problem is that sorkle is trying to sell snake oil on the name of Stanford and AI. Under the hood it is just a data generation pipeline. Remember you can't put label on random data. So "Not data generation. Label generation" is totally does not make any sense and sound to me like "brown sugar".

Re: Snorkel AI: Putting Data First in ML Development

#52

Post by ianhorn »

ajratner wrote: post #48
ianhorn wrote: post #11

I have a question for the snorkel folks if they're lurking here. With noisy label functions for large amounts of data, I could easily see the cases where the label functions fail correlating with classes that are already having a hard time (disadvantaged/underrepresented/marginalized groups, etc). Has there been any work on using these tools while making sure to avoid dangerous biases? It seems like the kind of tool…

Lurking for a few more min... great question! Dealing with both class imbalance and issues of pernicious biases in both underlying data distributions and training labels is an extremely important topic. Our underlying theory deals with local biases (e.g. individual labeling functions or sources of training signal being biased) but systemic biases (e.g. the user driving the system being biased) are certainly tougher.…

Thanks for the answer! The local vs systemic bias thing is particularly interesting for a system like snorkel. I have a clarifying question.

I'm imagining a dumb example like recipes, where "1 tsp salt" is a common format for ingredient. I'd imagine that the majority of ingredients follow that format, so it'd be a natural function to write. I'd also imagine that there's a correlation between following that format and being a recipe with a european background.

Generalize that a little bit, and almost by definition the simplest N rules that get the most coverage will cover the majority cases best. Being outside the majority cases is probably correlated with most "human issues," defined however you want. Being an artifact of the properties of what the simplest N rules cover, I'm not clear whether it'd be defined as local or systemic in the sense you've worked on.

I'm curious whether this falls under the theory you've worked on already or the theory you're talking about pursuing in the future. If it's something you've worked on already, I'd be very interested in reading what you have.

Re: Snorkel AI: Putting Data First in ML Development

#53

Post by eggie5 »

ZeroCool2u wrote: post #41
eggie5 wrote: post #34

Earlier quoted context omitted.

can you educate me on how you do multi-label w/ snorkel? As far as I can understand that's one it's largest drawbacks.

Fair question. I didn't personally supervise our last intern, it was my turn the summer before, so I'm not as deeply familiar with it. Now that you bring this up though, I think perhaps I may have misspoken. When I said muti-label, I think that was our goal originally, but because of the constraints of Snorkel you mentioned, we ended up reframing the problem into many single class models instead. They would both work…

yeah, you can do single-label w/ snorkel, but not multi-label. Multi-label snorkel would be the killer feature bc making the negatives (ie for a softmax) is very hard especially when you work w/ user-interaction systems with an unknown negative distribution.

Re: Snorkel AI: Putting Data First in ML Development

#54

Post by eggie5 »

ajratner wrote: post #47
eggie5 wrote: post #24

anyone figure out how to do multi-label classification w/ snorkel? It seems like it's current formulation only supports single-label, ie softmax. I find in practice, especially w/ user-interaction, system most problems are not single-label, but multi-label. Also, in the single-label setting it's often necessary to define an negative "OTEHR" class which is very difficult to define w/ snorkel in my experience.

Single label has been the applicable one for most of the applications we've tackled to date, but agreed that multi-label is also very important! More coming here soon...

Thanks Alex. I'm sure you can relate that when you have a an unbounded input distribution (like w/ user-interaction systems), defining that other class w/ current snorkel is difficult/impossible.

Re: Snorkel AI: Putting Data First in ML Development

#55

Post by woeirua »

ajratner wrote: post #45
woeirua wrote: post #39

I can see how this would work for tabular and text data, where the labeling functions are well-defined. I don't understand, at all , how this would work with computer vision tasks where heuristics are pretty much impossible to define. That said, I don't see anything here that would prevent you from using a pre-trained conv net as a labeling function, but I expect that multiple conv nets trained on a small corpus of d…

First: Snorkel Flow absolutely does not generalize to every ML problem :). IMO defining where different systems and approaches do and don't work best is one of the most important and most challenging problems in ML systems research- as noted, we've worked to give detail on this for Snorkel over the years... no perfect answers, but some notes below: - As you imply, a lot has to do with the available sources of input s…

Does it work for semantic segmentation? That's really where I'm struggling to see how this could work.

Re: Snorkel AI: Putting Data First in ML Development

#56

Post by woeirua »

ZeroCool2u wrote: post #42
woeirua wrote: post #39

I can see how this would work for tabular and text data, where the labeling functions are well-defined. I don't understand, at all , how this would work with computer vision tasks where heuristics are pretty much impossible to define. That said, I don't see anything here that would prevent you from using a pre-trained conv net as a labeling function, but I expect that multiple conv nets trained on a small corpus of d…

At the ODSC presentation I went to last year where the team presented they actually used a vision problem as their canonical example. It's hard to grasp without a concrete example, but it makes a lot of sense the way they explained it. For example, lets assume you want to identify something like a lung tumor. So you have many MRI images and they're all largely the same template of image. Using traditional image proce…

Yeah, but for some tasks, like recognizing a car, or something else you'd have to write pretty sophisticated code to even get to a reasonable result. It's far easier in those cases to use supervised learning to have a NN learn to do it for you.

Re: Snorkel AI: Putting Data First in ML Development

#57

Post by ajratner »

eggie5 wrote: post #53
ZeroCool2u wrote: post #41

Earlier quoted context omitted.

Fair question. I didn't personally supervise our last intern, it was my turn the summer before, so I'm not as deeply familiar with it. Now that you bring this up though, I think perhaps I may have misspoken. When I said muti-label, I think that was our goal originally, but because of the constraints of Snorkel you mentioned, we ended up reframing the problem into many single class models instead. They would both work…

yeah, you can do single-label w/ snorkel, but not multi-label. Multi-label snorkel would be the killer feature bc making the negatives (ie for a softmax) is very hard especially when you work w/ user-interaction systems with an unknown negative distribution.

You can always do multi-label as a multi-task learning model (or just a set of binary models), which is something we (and many others) have explored before! A lot of the adjustments for mainline Snorkel have to do with (A) the semantics of the labeling functions (need to be able to express that something is not class A and/or have a general per-class prior) and (B) all the infra to support what is just now a bunch of independent per-label binary tasks, at base

Re: Snorkel AI: Putting Data First in ML Development

#58

Post by ajratner »

eggie5 wrote: post #54
ajratner wrote: post #47

Earlier quoted context omitted.

Single label has been the applicable one for most of the applications we've tackled to date, but agreed that multi-label is also very important! More coming here soon...

Thanks Alex. I'm sure you can relate that when you have a an unbounded input distribution (like w/ user-interaction systems), defining that other class w/ current snorkel is difficult/impossible.

Yeah definitely- and would love to chat sometime, as this is a space I've at least had less direct hands-on interaction with. There's a line of work in the ML literature on "Positive unlabeled (PU) learning"--basically, setting where there are only positive labels or abstains--with a lot of theoretical ties to what our stuff rests on, I think a tie in here is interesting. Of course, most of these approaches rely on some (to varying degrees) hidden and very strong distributional assumption... anyway looking forward to a chat!

Re: Snorkel AI: Putting Data First in ML Development

#59

Post by ajratner »

gas9S9zw3P9c wrote: post #18
nobodywillobsrv wrote: post #17

What does it actually do? I know "label data" but anything can do that. Is it just a pipeline system with some helpers for running a couple of ML related functions? Is it UI based? Where do you run it? I know these areas well and got nearly nothing from reading the splash page on the site.

I also have problems understanding what exactly it does. I just briefly skimmed the paper, but it seems like the idea is as follows. Assuming you don't have ground truth labels for your data: 1. Generate many different nosiy labels for your data by writing functions. These don't need to be correct, but they should make uncorrelated errors. They are basically domain knowledge you have of your data. 2. Snorkel takes th…

Yup! The LFs can also actually be correlated- just not too correlated (roughly, think of needing at least three mostly uncorrelated cliques, to be precise).

Part of the high level description, though, is that a lot of different parts and lines of work are integrated into Snorkel Flow beyond just this original programmatic labeling idea. So also programmatic operators for data augmentation, "slicing" or partitioning of data, and the overall end-to-end platform (UI + SDK) supporting iterative development of ML models via this paradigm of programmatic training data.

Re: Snorkel AI: Putting Data First in ML Development

#60

Post by ajratner »

woeirua wrote: post #55
ajratner wrote: post #45

Earlier quoted context omitted.

First: Snorkel Flow absolutely does not generalize to every ML problem :). IMO defining where different systems and approaches do and don't work best is one of the most important and most challenging problems in ML systems research- as noted, we've worked to give detail on this for Snorkel over the years... no perfect answers, but some notes below: - As you imply, a lot has to do with the available sources of input s…

Does it work for semantic segmentation? That's really where I'm struggling to see how this could work.

More advanced structured prediction tasks are still definitely on the cutting edge- mainly IMO down to defining the semantics of the programmatic user input like labeling functions for these kinds of tasks. Some recent work (http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf) has extended these semantics for sequence tagging, as an example- so some exciting moves in this direction!
Post reply on HN