Dataturks – ML data annotations and labeling doesn't need to suck

There are loads of tools for this, depending on what you need to label (rectangles, tagging, polygons, etc).

Here's one I wrote for my own use: https://github.com/jveitchmichaelis/deeplabel

It's not terribly feature complete (and it's not stable), but it's functional and pretty fast for basic bounding box annotation. My aim was to minimise clicks compared tools like labelimg, and also to have a database that I could work on with changes being stored immediately. Other tools seem to forget where you left off. I also didn't want a dependency on Python or the cloud.

It runs on a sqlite database, so presumably in a shared environment you could actually use it real-time collaboratively. I was doing some labelling for a project with my girlfriend and we simply took turns with the db and images a shared folder in Dropbox.

We used Sloth for segmentation labelling (polygons instead of pixels). Direct pixel labelling is very expensive, so usually you can save lots of time by simplifying the problem - one of them being polygon labelling.

Although Sloth is a gtk app it was the best option available year ago - much better than free web based options available back then. We packaged it into single windows exe that we were distributing to labellers in a zip alongside data to label. Labellers would send back zip + annotated data in json format custom to Sloth. Zips landed in s3 folder and we had some scripts doing analytics on that (e.g. Labellers reliability). It was very easy to extend and add new functions to sloth. Although it seems low tech comparing to web labelling it took us surprisingly lots of work to beat this workflow with web based purpose-built workflow.

Hey, Thanks for checking us out and the inputs. API access is getting ready to launch as I am typing this. Agree that rater reliability is a very important thing which will matter here, ultimately it converts to dataset quality. Over the period of time, we would show dataset quality in some form using feedbacks from dataset users.

Hey, If its a free plan, data stays open to public. But for paid [for larger & enterprise datasets] plans, DataTurks doesn't have any ownership of data, and we don't keep any copies.