Live data from Hacker News

Prodigy: A new tool for radically efficient machine teaching

explosion.ai

51–60 of 75 posts

Re: Prodigy: A new tool for radically efficient machine teaching

#51
post #24

Guess the radical efficiency didn't carry over to their web server

> spaCY, the leading open-source NLP tool? Sounds like marketing BS. what about OpenNLP and Stanford's for NLP?

They have some hard comparisons in some of their earlier blog posts on how spacy compares to the other popular open source NLP libraries. In my experience it has been much easier to use and faster than things like Stanford's library or NLTK. In general it's aimed at production or commercial use, whereas the other libraries I typically hear mentioned are aimed at a more academic audience.

Re: Prodigy: A new tool for radically efficient machine teaching

#52
Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts.

1. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in contrast to something like the VGG annotations system: http://www.robots.ox.ac.uk/~vgg/software/via/via.html

2. I don't see an actual annotations capability for images in the demo. Not sure if that is just a pretotype page, but IMO image classification/segmentation is the place where this tool would really benefit the community.

3. It's unclear to me how or if I retrieve my trained model or even just the annotated structure (.csv?, .json?) from this system. Do I get a .pb somehow that I can import into TF or am I locked into an API with my new model served from Prodigy? My guess would be the latter.

I think what this wants to be is a human validation system for training, which also improves the Prodigy nets through crowd sourcing. Definitely a win-win in the short term, but it has the limitations of the initial model and the ability for the user/client to tweak the system and output the results.

Matroid is doing something similar here, but I have been unimpressed with their offering so far.

Re: Prodigy: A new tool for radically efficient machine teaching

#53

Earlier quoted context omitted.

spaCY, the leading open-source NLP tool? Agreed, the description is definitely cringe-worthy. As if whoever wrote that wasn't aware that these are language geeks they're marketing to.

Self-respecting language geeks keep up with the times. What's your case for "leading open-source"? Here's a look at Spacy blowing Stanford Core NLP out of the water (via github stars, you can take a look at commits and more from the same tool): https://www.datascience.com/trends?trends=4812,7214,7165&tre...

When a new tool is announced, there is a lot of casual interest. Hence the 'explosion' in the beginning. After some time, things reach steady-state, and you can see that Spacy's interest is starting to fall below Core NLP's in the last few weeks.

In other words: Spacy is sinking.

Re: Prodigy: A new tool for radically efficient machine teaching

#54

Earlier quoted context omitted.

spaCY, the leading open-source NLP tool? Agreed, the description is definitely cringe-worthy. As if whoever wrote that wasn't aware that these are language geeks they're marketing to.

Self-respecting language geeks keep up with the times. What's your case for "leading open-source"? Here's a look at Spacy blowing Stanford Core NLP out of the water (via github stars, you can take a look at commits and more from the same tool): https://www.datascience.com/trends?trends=4812,7214,7165&tre...

Actually I don't follow any of these tools closely to know whether they're currently "leading" or not.

It's just that, wording-wise "the leading open source X" exudes marketing-speak, which I find language geeks tend to have robust antibodies against.

This kind of lingo work (sort of) for the market, say, MongoDB is in. But for the users of these tools, I suspect not so much.

Re: Prodigy: A new tool for radically efficient machine teaching

#55

Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts. 1. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in c…

Thanks for the engaging questions! Reading between the lines, I think there's an important point that hasn't come across. Prodigy isn't SaaS --- it's a library you download and run. You can extend and customise every aspect of it, and there's definitely no lock-in. The model (and annotations) never have to leave your servers.

For the specific questions:

1. The built-in web views all have binary annotation interfaces. This is more of a design choice than a fundamental limitation, and the front-end is extensible --- you can add your own web views if you need to.

The binary interface is sort of a position statement. We think this is The Way, so we want you to try it. We'll have more input components in future, but at the start we want to guide people towards the intended workflow.

2. The beta focuses on NLP support, but there's a front-end for image classification, and a workflow page: https://prodi.gy/docs/workflow-image-classification .

3. You can usually get some accuracy improvement by retraining once all the annotations are available. I've not found a streaming SGD algorithm that works as well as the simple iterate-and-shuffle batch process. Batch training also lets you tune the hyper-paramers. You can read more about this here: https://prodi.gy/docs/workflow-named-entity-recognition#trai...

I would suggest writing a Prodigy recipe to do the batch training. That way you can pass in the dataset ID, instead of exporting the annotations. There's no problem with exporting the annotations and running a script, though. Again --- it's all on your computer. You can run it however you like.

Re: Prodigy: A new tool for radically efficient machine teaching

#56

So this is just fluff?

No, I don't think so. People tend to over-emphasize the latest ML techniques when the greatest improvements right now come from better and cleaner data sets. I am interested in this sort of thing because we were just about to try and build something like it ourselves.

Re: Prodigy: A new tool for radically efficient machine teaching

#58

I work with data as a neuroscientist, but I haven't used ML. What is an an annotation in this context?

Adding mark-up, basically.

Examples for text data: topic tags, marking mentions of companies, finding descriptions of protein interactions. For images people mostly do segmentation (finding object boundaries) and classification.

If the model is generating images or text, you usually need to do annotation to evaluate the model. For instance, you need to know whether the translations are grammatical, whether an image is coloured correctly, etc.

Re: Prodigy: A new tool for radically efficient machine teaching

#59
post #50

Looks very nice, although it always takes me a bit to figure out what they're talking about with these sorts of things because I have to remind myself that most ML/DL stuff is supervised. What I research is unsupervised. They kind of have this weird dissing of unsupervised scenarios, though. It's not like supervised or unsupervised is better or worse, they're just surrounding different problems. They can talk up thei…

I see how that came across as obnoxious, so thanks for the perspective.

I do think there's a pretty common failure mode for teams who don't have much experience with ML, though. Teams who don't have much experience with ML often take "We don't have much data" as a parameter of their problem, and don't see that this is something they can decide to change. This can lead to a lot of time spent experimenting with different unsupervised approaches that are a poor fit for what they're trying to do.

Re: Prodigy: A new tool for radically efficient machine teaching

#60
post #24

Guess the radical efficiency didn't carry over to their web server

> spaCY, the leading open-source NLP tool? Sounds like marketing BS. what about OpenNLP and Stanford's for NLP?

I have to agree. Explosion/Spacy has a lot of Marketing BS. That said I think SpaCy is actually pretty solid and when you are in the NLP field give it a try.
Post reply on HN