Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Mechanical Turk is twenty years old today

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Tell HN: Mechanical Turk is twenty years old today

#11

Post by mtlynch »

I'm a software developer, but I took a brief career break in 2011 to try B2B sales for an ISP. I was the only sales rep with experience as a developer, so I was always looking for ways to use my software skills to get an edge as a sales rep.

The most valuable prospects were businesses in buildings where we had a direct fiber connection. There were sites online that purported to list the buildings and leads that the company bought from somewhere, but the sources were all really noisy. Like 98% of the time, the phone number was disconnected or didn't match the address the source said, so basically nobody used these sources.

I thought MTurk would be my secret weapon. If I could pay someone like $0.10/call to call business and confirm the business name and address, then I'd turn these garbage data sources into something where 100% of the prospects were valid, and none of the sales reps competing with me would have time to burn through these low-probability phone numbers.

The first day, I was so excited to call all the numbers that the MTurk workers had confirmed, and...

The work was all fake. They hadn't called anyone at all. And they were doing the jobs at like 4 AM local time when certainly nobody was answering phones at these businesses.

I tried a few times to increase the qualifications and increase pay, but everyone who took the job just blatantly lied about making the calls and gave me useless data.

Still, I thought MTurk was a neat idea and wish I'd found a better application for it.

Re: Tell HN: Mechanical Turk is twenty years old today

#12

Post by frmersdog »

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

Re: Tell HN: Mechanical Turk is twenty years old today

#13

Post by maxrmk »

frmersdog wrote: post #12

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

What do you see as net negative about it? I’m familiar with the product but not that aware of how it’s been used.

Re: Tell HN: Mechanical Turk is twenty years old today

#14

Post by edoceo »

frmersdog wrote: post #12

If there's any justice, a good number of comments will focus on the ethical nightmare MTurk turned out to be. Apologies to the people who worked on it, but it's fair and appropriate for observers to point out when someone has spent their time and energy creating something that is a net negative for the state of society. That's probably the case here.

These are extraordinary claims (yea?). I'm sure there are great stories of opportunity creation and destruction - how could we even measure the net effect?

Re: Tell HN: Mechanical Turk is twenty years old today

#15

Post by deadbabe »

With LLMs, I think we will finally have the missing piece needed to make something like MTurk work at scale.

Bad data or false work was a big problem on MTurk, but now LLMs should be able to act as reasonable quality assurance for each and every piece of work a worker commits. The workers can be ranked and graded based on the quality of their work, instantly, instead of requiring human review.

You can also flip the model and have LLMs do the unit of work, and have humans as a verification layer, and the human review sanity checked again by an LLM to ensure people aren’t just slacking off and rubber stamping everything. You can easily do this by inserting blatantly bad data at some points and seeing if the workers pick up on it. Fail the people who are letting bad data pass through.

For a lot of people, I think this will be the future of work. People will go to schools to get rounded educations and get degrees in “Human Cognitive Tasks” which makes them well suited for doing all kinds of random stuff that fills in gaps for AI. Perhaps they will also minor in some niche fields for specific industries. Best of all, they can work their own hours and at home.

Re: Tell HN: Mechanical Turk is twenty years old today

#17

Post by hiddencost »

stevejb wrote: post #10

Using the Propser.com data set (a peer-to-peer lending market), I used MTurk to analyze the images of people applying for a loan. This was used in a finance research project with 3 University of Washington professors of Finance. The idea was that the Prosper data set contained all of the information that a lending officer would have, but they also had user-submitted pictures. We wanted to see if there was value in th…

Yikes. Have you ever considered that you were hurting people?

Re: Tell HN: Mechanical Turk is twenty years old today

#18

Post by hiddencost »

deadbabe wrote: post #15

With LLMs, I think we will finally have the missing piece needed to make something like MTurk work at scale. Bad data or false work was a big problem on MTurk, but now LLMs should be able to act as reasonable quality assurance for each and every piece of work a worker commits. The workers can be ranked and graded based on the quality of their work, instantly, instead of requiring human review. You can also flip the m…

I, uh, you do understand the causality issues here? I'm reminded of the onion headline "Tab of LSD feeling a lot of pressure from tech worker to come up with new ideas"

Re: Tell HN: Mechanical Turk is twenty years old today

#20

Post by danpalmer »

I have looked at MTurk many times throughout my career. In particular my previous company had a lot of data cleaning, scraping, product tagging, image description, and machine learning built on these. This was all pre-LLM. MTurk always felt like it would be a great solution.

But every time I looked at it I persuaded myself out of it. The docs really down played the level of critical thinking that we could expect, they made it clear that you couldn't trust any result to even human-error levels, you needed to test 3-5 times and "vote". You couldn't really get good results for unstructured outputs instead it was designed around classification across a small number of options. The bidding also made pricing it out hard to estimate.

In the end we hired a company that sat somewhere between MTurk and fully skilled outsourcing. We trained the team in our specific needs and they would work through data processing when available, asking clarifying questions on Slack, and would reference a huge Google doc that we had with various disambiguations and edge cases documented. They were excellent. More expensive that MTurk on the surface, but likely cheaper in the long run because the results were essentially as correct as anyone could get them and we didn't need to check their work much.

In this way I wonder if MTurk never found great product market fit. It languished in AWS's portfolio for most of 20 years. Maybe it was just too limited?

Post reply on HN