Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We in-housed our data labelling

ericbutton.co

Post reply on HN
Print view View original

41–50 of 51 posts

Re: We in-housed our data labelling

#41

Post by kristian1109 »

stevage wrote: post #32
yorwba wrote: post #18

> Failing a test will cost a user 600 points, or roughly the equivalent of 15 minutes of work on the platform. A correctly tuned penalty system removes the need for setting reviewer accuracy minimums; poor performers will simply not earn enough money to continue on the platform. This still sets a reviewer accuracy minimum, but it is determined implicitly by the arbitrary test penalty instead of consciously chosen bas…

The screenshot shows that 25,000 points is about $50, so 500 points is about $1. If 600 points is about 15 minutes work, that means reviewers are getng paid less than $4 per hour?

All the numbers in the article are made up (I coded up a quick JSON so we could render the page without showing real user information). Our reviewers make ~10x that number. In hindsight, we shouldn't have made up numbers that ended up with this math. That's on me!

Re: We in-housed our data labelling

#42

Post by kristian1109 »

gpvos wrote: post #20

> All labellers are either licensed pilots or controllers (or VATSIM pilots/controllers). I would think such people can make better money by actually working as a pilot or controller?

For sure they can! The vast majority of our reviewers make a few extra $hundred every week around their day jobs in aviation. Our target audience for this is the folks who enjoy spending their free time watching YouTube videos like VASAviation or PilotDebrief (like me). We get a kick out of listening to air traffic control audio and hearing what's going on in the airspace system.

Re: We in-housed our data labelling

#43

Post by EricButton »

stevage wrote: post #31

So they are building a system which has all the hallmarks of an extremely addictive game. But that's ok because they pay the players a small amount of money? They didn't even address the wellbeing of players, managing addiction and overwork etc.

Some context, the average labeller on our platform puts in a single digit number of hours on our platform, works whenever and how much they want, and earn significantly more than an uber driver.

Re: We in-housed our data labelling

#44

Post by stevage »

kristian1109 wrote: post #41
stevage wrote: post #32

Earlier quoted context omitted.

The screenshot shows that 25,000 points is about $50, so 500 points is about $1. If 600 points is about 15 minutes work, that means reviewers are getng paid less than $4 per hour?

All the numbers in the article are made up (I coded up a quick JSON so we could render the page without showing real user information). Our reviewers make ~10x that number. In hindsight, we shouldn't have made up numbers that ended up with this math. That's on me!

Ah, cool. But that also means each penalty is $10. Eep.

Re: We in-housed our data labelling

#45

Post by stevage »

EricButton wrote: post #43
stevage wrote: post #31

So they are building a system which has all the hallmarks of an extremely addictive game. But that's ok because they pay the players a small amount of money? They didn't even address the wellbeing of players, managing addiction and overwork etc.

Some context, the average labeller on our platform puts in a single digit number of hours on our platform, works whenever and how much they want, and earn significantly more than an uber driver.

Thanks.

To play devil's advocate, the average gambler is not problematic either, it's the outliers that are the problem.

Re: We in-housed our data labelling

#46

Post by wodenokoto »

neilv wrote: post #36
wodenokoto wrote: post #14

Earlier quoted context omitted.

What do you mean “didn’t age well”, its a brand new article. It hasn’t aged at all.

The article is dated the 16th, before the HN outrage around the 25th, over the video skit about some YC factory worker surveillance startup. If the writer had the benefit of seeing the few-post outrage on the 25th, they probably would've written the article differently, and maybe also reflected on the dynamic with the workers. In a startup, when you have to do all the things, and you're constantly learning, it's easy…

I don’t think HN being discontented about the gig economy is anything new, at least not if you are counting new in weeks and not years.

Re: We in-housed our data labelling

#47

Post by EricButton »

stevage wrote: post #45
EricButton wrote: post #43

Earlier quoted context omitted.

Some context, the average labeller on our platform puts in a single digit number of hours on our platform, works whenever and how much they want, and earn significantly more than an uber driver.

Thanks. To play devil's advocate, the average gambler is not problematic either, it's the outliers that are the problem.

fair point!

Re: We in-housed our data labelling

#48

Post by collingreen »

ellisv wrote: post #21
jsemrau wrote: post #4

Earlier quoted context omitted.

What would a product look like in this space?

There are several data labeling products on the market such as Label Studio. I’ve resorted to building my own annotation apps.

I ALSO have resorted to building my own labeling even though there are great generic labeling tools out there. I think this is a missing piece of the landscape but I don't know enough about the space yet to say what the solution should be.

Re: We in-housed our data labelling

#49

Post by almostgotcaught »

antognini wrote: post #5
xnx wrote: post #3

Earlier quoted context omitted.

Good data and good evals are two legs of the 3-legged stool that a lot of AI teams are missing.

It also can't really be overstated how helpful it is as an ML engineer to simply spend the time going through thousands of examples yourself. If you abstract yourself away from the data and just "make metric go up" you'll be missing out on valuable insights about how and why your model might be failing.

It's almost as if (bear with me ...) these "artificial intelligences" actually need "human intelligences" to guide them. Maybe we can think up a "system" where "experts" can codify rules for the "artificial intelligence" to follow.

Ok the sarcasm got too thick but my point is if the engineer has to spend the time to comb thousands of examples then you don't have AI you have a man in a box pretending to be a machine that plays chess.

Re: We in-housed our data labelling

#50

Post by llm_trw »

almostgotcaught wrote: post #49
antognini wrote: post #5

Earlier quoted context omitted.

It also can't really be overstated how helpful it is as an ML engineer to simply spend the time going through thousands of examples yourself. If you abstract yourself away from the data and just "make metric go up" you'll be missing out on valuable insights about how and why your model might be failing.

It's almost as if (bear with me ...) these "artificial intelligences" actually need "human intelligences" to guide them. Maybe we can think up a "system" where "experts" can codify rules for the "artificial intelligence" to follow. Ok the sarcasm got too thick but my point is if the engineer has to spend the time to comb thousands of examples then you don't have AI you have a man in a box pretending to be a machine t…

We have human teachers for much the same reasons.

Are humans just other humans hiding in boxes pretending to play chess?

Post reply on HN