Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Mechanical Turk is twenty years old today

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Tell HN: Mechanical Turk is twenty years old today

#31

Post by akerl_ »

hiddencost wrote: post #17
stevejb wrote: post #10

Using the Propser.com data set (a peer-to-peer lending market), I used MTurk to analyze the images of people applying for a loan. This was used in a finance research project with 3 University of Washington professors of Finance. The idea was that the Prosper data set contained all of the information that a lending officer would have, but they also had user-submitted pictures. We wanted to see if there was value in th…

Yikes. Have you ever considered that you were hurting people?

The paper is a study of an existing market. They looked at data about people who had requested loans and data about which of those loans were funded, with the intent of seeing whether or not lenders were being biased by requester photos. They found that they were.

Say more about how studying that bias is hurting people?

Re: Tell HN: Mechanical Turk is twenty years old today

#32

Post by rgovostes »

I was working for a marine biologist applying computer vision to the task of counting migrating river herring. The data set was lousy, with background movement, poor foreground/background separation, inconsistent lighting, and slow exposures that created blurry streaks. Moreover, the researchers had posted the frames to some citizen science platform and asked volunteers to tag each fish with a single point—sufficient for counting by hand, but basically useless for object detection.

In desperation I turned to Mechanical Turk to have the labels redone as bounding boxes, with some amount of agreement between labelers. Even then, results were so-so, with many workers making essentially random labels. I had to take another pass, flipping rapidly through thousands of frames with low confidence scores, which gave me nausea not unlike seasickness.

Re: Tell HN: Mechanical Turk is twenty years old today

#33

Post by xwowsersx »

hiddencost wrote: post #17
stevejb wrote: post #10

Using the Propser.com data set (a peer-to-peer lending market), I used MTurk to analyze the images of people applying for a loan. This was used in a finance research project with 3 University of Washington professors of Finance. The idea was that the Prosper data set contained all of the information that a lending officer would have, but they also had user-submitted pictures. We wanted to see if there was value in th…

Yikes. Have you ever considered that you were hurting people?

How so? Read the paper. The methodology was entirely observational. They did not intervene in the prosper.com loan market or interact with the borrowers. If anything, the paper identified a form of bias that exists in the real world, namely that people commonly "perceived" as less trustworthy are penalized despite their actual creditworthiness.

Re: Tell HN: Mechanical Turk is twenty years old today

#34

Post by [deleted] »

social_quotient wrote: post #30

I’ve run millions of jobs on MTurk. For a major mall operator in the USA, we had an issue with tenants keeping their store hours in sync between the mall site and their own site. So we deployed MTurk workers in redundant multiples for each retail listing… 22k stores at the time, checked weekly from October through mid-January. Another use case.. figuring out whether a restaurant had OpenTable as an option. This also…

[deleted]

Re: Tell HN: Mechanical Turk is twenty years old today

#35

Post by electroly »

Several times, I had MTurk workers transcribe a yearly printed pricing catalog that was a boon to our small business. Inconsistently-structured tabular data intermixed with pictures that OCR of the day did a terrible job with.

Later, we needed to choose the best-looking product picture from a series of possible pictures (collected online) for every SKU, for use in our website's inventory browser. MTurk to the rescue--their human taste was perfect, and it was effortless on my part.

Neither of these were earthshattering from a tech perspective, and I'm sure these days AI could do it, but back then MTurk was the perfect solution. Humans make both random and consistent errors and it was kinda fun to learn how to deal with both kinds of error. I learned lots of little tricks to lower the error rate. As a rule, I always paid out erroneous submissions (you can choose to reject them but it's easier to just pay for all submissions) and just worked to improve my prompts. I never had anyone maliciously or intentionally try to submit incomplete or wrong work, but lots of "junk" happens with the best of intentions.

Re: Tell HN: Mechanical Turk is twenty years old today

#38

Post by pvankessel »

social_quotient wrote: post #30

I’ve run millions of jobs on MTurk. For a major mall operator in the USA, we had an issue with tenants keeping their store hours in sync between the mall site and their own site. So we deployed MTurk workers in redundant multiples for each retail listing… 22k stores at the time, checked weekly from October through mid-January. Another use case.. figuring out whether a restaurant had OpenTable as an option. This also…

The automation one is so true! When I first deployed a huge job to MTurk, with so much money on the line I wanted to be careful, and I wrote some heuristics to auto-ban Turkers who worked their way through the HITs suspiciously quickly (2 standard deviations above the norm, iirc) - and damn did I wake up to a BUNCH of angry (but kind) emails. Turns out, there was a popular hotkey programming tool that Turk Masters made use of to work through the more prized HITs more efficiently, and on one of their forums someone shared a script for ours. I checked their work and it was quality, they were just hyper-optimizing. It was reassuring to see how much they cared about doing a good job.
Post reply on HN