Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Mechanical Turk is twenty years old today

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 65 posts

Re: Tell HN: Mechanical Turk is twenty years old today

#21

Post by cactusplant7374 »

comrade1234 wrote: post #4

My wife had dozens, well probably over 100, handwritten recipes from a dead relative. They were pretty difficult to read. I scanned them and used mturk to have them transcribed. Most of the work was done by one person - i think she was a woman in the Midwest, it's been like 15-years so the details are hazy. A few recipes were transcribed by people overseas but they didn't stick at it. I had to reject only one transcr…

Do you think ChatGPT could do the same work now? It would be interesting to try it.

Re: Tell HN: Mechanical Turk is twenty years old today

#22

Post by crossbody »

frmersdog wrote: post #12

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

"probably". Care to provide reasoning or is this just a knee jerk reaction? Are you familiar with the service and how it works?

Re: Tell HN: Mechanical Turk is twenty years old today

#23

Post by pvankessel »

I used MTurk heavily in its hey-day for data annotation - it was an invaluable tool for collecting training data for large-scale research projects, I honestly have to credit it with enabling most of my early career triumphs. We labeled and classified hundreds of thousands of tweets, Facebook posts, news articles, YouTube videos - you name it. Sure, there were bad actors who gave us fake data, but with the right qualifications and timing checks, and if you assigned multiple Turkers (3-5) to each task, you could get very reliable results with high inter-rater reliability that matched that of experts. Wisdom of the crowd, or the law of averages, I suppose. Paying a living wage also helped - the community always got extremely excited when our HITs dropped and was very engaged, I loved getting thank yous and insightful clarifying questions in our inbox. For most of this kind of work, I now use AI and get comparable results, but back in the day, MTurk was pure magic if you knew how to use it to its full potential. Truthfully I really miss it - hitting a button to launch 50k HITs and seeing the results slowly pour in overnight (and frantically spot-checking it to make sure you weren't setting $20k on fire) was about as much of a rush as you can get in the social science research world.

Re: Tell HN: Mechanical Turk is twenty years old today

#25

Post by larodi »

frmersdog wrote: post #12

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

Happily I then can state we did create nothing based on MTurk as it had this negative ethical side to it from day one.

Re: Tell HN: Mechanical Turk is twenty years old today

#26

Post by fsniper »

cactusplant7374 wrote: post #21
comrade1234 wrote: post #4

My wife had dozens, well probably over 100, handwritten recipes from a dead relative. They were pretty difficult to read. I scanned them and used mturk to have them transcribed. Most of the work was done by one person - i think she was a woman in the Midwest, it's been like 15-years so the details are hazy. A few recipes were transcribed by people overseas but they didn't stick at it. I had to reject only one transcr…

Do you think ChatGPT could do the same work now? It would be interesting to try it.

I used Gemini to decode and transcribe an old (and well known) cursive hand written mail. I couldn't read it at all. It managed to do this in a few seconds. I am not sure if it used an already available transcription or not. However if not, it was amazing work.

Re: Tell HN: Mechanical Turk is twenty years old today

#27

Post by akerl_ »

maxrmk wrote: post #13
frmersdog wrote: post #12

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

What do you see as net negative about it? I’m familiar with the product but not that aware of how it’s been used.

It's basically a way for people to externalize tasks that require a human but pay fractions of what it would cost to actually employ those humans.

Mechanical Turk was one of the early entrants into "how can we rebrand outsourcing low skill labor to impoverished people and pay them the absolute bare minimum as the gig economy".

Re: Tell HN: Mechanical Turk is twenty years old today

#29

Post by amelius »

akerl_ wrote: post #27
maxrmk wrote: post #13

Earlier quoted context omitted.

What do you see as net negative about it? I’m familiar with the product but not that aware of how it’s been used.

It's basically a way for people to externalize tasks that require a human but pay fractions of what it would cost to actually employ those humans. Mechanical Turk was one of the early entrants into "how can we rebrand outsourcing low skill labor to impoverished people and pay them the absolute bare minimum as the gig economy".

Yes, and "use the output of MTurk workers to make themselves redundant."

Re: Tell HN: Mechanical Turk is twenty years old today

#30

Post by social_quotient »

I’ve run millions of jobs on MTurk.

For a major mall operator in the USA, we had an issue with tenants keeping their store hours in sync between the mall site and their own site. So we deployed MTurk workers in redundant multiples for each retail listing… 22k stores at the time, checked weekly from October through mid-January.

Another use case.. figuring out whether a restaurant had OpenTable as an option. This also changes from time to time, so we’d check weekly via MTurk. 52 weeks a year across over 100 malls. Far fewer in quantity, think 2-300. But it’s still more work than you’d want to staff.

A fun more nuanced use case: In retail mall listings, there’s typically a link to the retailer’s website. For GAP, no problem… it’s stable. But for random retailers (think kiosk operators), sometimes they’d lose their domain, which would then get forwarded to an adult site. The risk here is extremely high. So daily we would hit all retailer website links to determine if they contained adult or objectionable content. If flagged, we’d first send to MTurk for confirmation, then to client management for final determination. In the age of AI this would be very different, but the number of false positives was comical. Take a typical lingerie retailers and send it to a skin detection algorithm… you’d maybe be surprised how many common retailers have NSFW homepages.

Now some pro tips I’ll leave you with.

- Any job worth doing on mturk is worth paying a decent amount of money for.

- never runs. Job 1 tile run it 3-5 times and then build a consensus algo on the results to get confidence

- assume they will automate things you would not have assumed automated - And be ready to get some junk results at scale

- think deeply on the flow and reduce the steps as much as possible.

- similar to how I manage Ai now. Consider how you can prove they did the work if you needed a real human and not an automation.

Post reply on HN