Guess the radical efficiency didn't carry over to their web server
> spaCY, the leading open-source NLP tool? Sounds like marketing BS. what about OpenNLP and Stanford's for NLP?
Prodigy: A new tool for radically efficient machine teaching
51–60 of 75 posts
Re: Prodigy: A new tool for radically efficient machine teaching
#521. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in contrast to something like the VGG annotations system: http://www.robots.ox.ac.uk/~vgg/software/via/via.html
2. I don't see an actual annotations capability for images in the demo. Not sure if that is just a pretotype page, but IMO image classification/segmentation is the place where this tool would really benefit the community.
3. It's unclear to me how or if I retrieve my trained model or even just the annotated structure (.csv?, .json?) from this system. Do I get a .pb somehow that I can import into TF or am I locked into an API with my new model served from Prodigy? My guess would be the latter.
I think what this wants to be is a human validation system for training, which also improves the Prodigy nets through crowd sourcing. Definitely a win-win in the short term, but it has the limitations of the initial model and the ability for the user/client to tweak the system and output the results.
Matroid is doing something similar here, but I have been unimpressed with their offering so far.
Re: Prodigy: A new tool for radically efficient machine teaching
#53Earlier quoted context omitted.
spaCY, the leading open-source NLP tool? Agreed, the description is definitely cringe-worthy. As if whoever wrote that wasn't aware that these are language geeks they're marketing to.
Self-respecting language geeks keep up with the times. What's your case for "leading open-source"? Here's a look at Spacy blowing Stanford Core NLP out of the water (via github stars, you can take a look at commits and more from the same tool): https://www.datascience.com/trends?trends=4812,7214,7165&tre...
In other words: Spacy is sinking.
Re: Prodigy: A new tool for radically efficient machine teaching
#54Earlier quoted context omitted.
spaCY, the leading open-source NLP tool? Agreed, the description is definitely cringe-worthy. As if whoever wrote that wasn't aware that these are language geeks they're marketing to.
Self-respecting language geeks keep up with the times. What's your case for "leading open-source"? Here's a look at Spacy blowing Stanford Core NLP out of the water (via github stars, you can take a look at commits and more from the same tool): https://www.datascience.com/trends?trends=4812,7214,7165&tre...
It's just that, wording-wise "the leading open source X" exudes marketing-speak, which I find language geeks tend to have robust antibodies against.
This kind of lingo work (sort of) for the market, say, MongoDB is in. But for the users of these tools, I suspect not so much.
Re: Prodigy: A new tool for radically efficient machine teaching
#55Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts. 1. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in c…
For the specific questions:
1. The built-in web views all have binary annotation interfaces. This is more of a design choice than a fundamental limitation, and the front-end is extensible --- you can add your own web views if you need to.
The binary interface is sort of a position statement. We think this is The Way, so we want you to try it. We'll have more input components in future, but at the start we want to guide people towards the intended workflow.
2. The beta focuses on NLP support, but there's a front-end for image classification, and a workflow page: https://prodi.gy/docs/workflow-image-classification .
3. You can usually get some accuracy improvement by retraining once all the annotations are available. I've not found a streaming SGD algorithm that works as well as the simple iterate-and-shuffle batch process. Batch training also lets you tune the hyper-paramers. You can read more about this here: https://prodi.gy/docs/workflow-named-entity-recognition#trai...
I would suggest writing a Prodigy recipe to do the batch training. That way you can pass in the dataset ID, instead of exporting the annotations. There's no problem with exporting the annotations and running a script, though. Again --- it's all on your computer. You can run it however you like.
Re: Prodigy: A new tool for radically efficient machine teaching
#56So this is just fluff?
Re: Prodigy: A new tool for radically efficient machine teaching
#57Re: Prodigy: A new tool for radically efficient machine teaching
#58I work with data as a neuroscientist, but I haven't used ML. What is an an annotation in this context?
Examples for text data: topic tags, marking mentions of companies, finding descriptions of protein interactions. For images people mostly do segmentation (finding object boundaries) and classification.
If the model is generating images or text, you usually need to do annotation to evaluate the model. For instance, you need to know whether the translations are grammatical, whether an image is coloured correctly, etc.
Re: Prodigy: A new tool for radically efficient machine teaching
#59Looks very nice, although it always takes me a bit to figure out what they're talking about with these sorts of things because I have to remind myself that most ML/DL stuff is supervised. What I research is unsupervised. They kind of have this weird dissing of unsupervised scenarios, though. It's not like supervised or unsupervised is better or worse, they're just surrounding different problems. They can talk up thei…
I do think there's a pretty common failure mode for teams who don't have much experience with ML, though. Teams who don't have much experience with ML often take "We don't have much data" as a parameter of their problem, and don't see that this is something they can decide to change. This can lead to a lot of time spent experimenting with different unsupervised approaches that are a poor fit for what they're trying to do.
Re: Prodigy: A new tool for radically efficient machine teaching
#60Guess the radical efficiency didn't carry over to their web server
> spaCY, the leading open-source NLP tool? Sounds like marketing BS. what about OpenNLP and Stanford's for NLP?