Live data from Hacker News

We in-housed our data labelling

ericbutton.co

1–10 of 51 posts

Re: We in-housed our data labelling

#2
Data is king. Even when a new better model comes along a high quality dataset is still just as valuable.

Paying top performers above market rates to do nothing but data labelling is a moat that just keeps getting deeper.

Re: We in-housed our data labelling

#3
post #2

Data is king. Even when a new better model comes along a high quality dataset is still just as valuable. Paying top performers above market rates to do nothing but data labelling is a moat that just keeps getting deeper.

Good data and good evals are two legs of the 3-legged stool that a lot of AI teams are missing.

Re: We in-housed our data labelling

#4
post #3
post #2

Data is king. Even when a new better model comes along a high quality dataset is still just as valuable. Paying top performers above market rates to do nothing but data labelling is a moat that just keeps getting deeper.

Good data and good evals are two legs of the 3-legged stool that a lot of AI teams are missing.

What would a product look like in this space?

Re: We in-housed our data labelling

#5
post #3
post #2

Data is king. Even when a new better model comes along a high quality dataset is still just as valuable. Paying top performers above market rates to do nothing but data labelling is a moat that just keeps getting deeper.

Good data and good evals are two legs of the 3-legged stool that a lot of AI teams are missing.

It also can't really be overstated how helpful it is as an ML engineer to simply spend the time going through thousands of examples yourself. If you abstract yourself away from the data and just "make metric go up" you'll be missing out on valuable insights about how and why your model might be failing.

Re: We in-housed our data labelling

#6
I think this didn't age well, for HN, and it prompts some serious questions about our techbro startup culture.

> Obvious but necessary: to incentivize productive work, we tie compensation to the number of characters transcribed, and assess financial penalties for failed tests (more on tests below). Penalties are priced such that subpar performance will result in little to no earnings for the labeller.

So, these aren't employees? The writeup talks about not trusting gig workers, but it sounds like they have gig workers, and a particularly questionable kind.

Not like independent contractors with the usual freedoms. But rather, under a punishing set of Kafkaesque rules, like someone was thinking only of computer programs, oops. "Gamified", with huge negative points penalties and everything. To be under threat of not getting paid at all.

I see that this article is dated the 16th, so it's before the HN outrage last week, over the founders who demoed a system for monitoring factory worker performance, and were ripped a new one online for dehumanizing employees.

Despite the factory system being not as invasive, dehumanizing, and potentially labor law-violating as what's described in this article: about whip-cracking of gig workers, moment-to-moment, and even not paying them.

I'm not even sure you'd get away with calling them "independent contractors", under these conditions, when workers save copies of this blog post, to show to labor lawyers and state regulators.

(Incidentally, I wasn't aware that a company working in aviation gets skilled workers this way. The usual way I've seen is to hire someone, with all the respect, rights, and benefits that entails. Or to hire a consultant who is decidedly not treated like a gig worker in a techno-dystopian sweatshop.)

I don't want Internet mob justice here, but I want to ask who is advising these startups regarding how they think of their place in the world, relative to other humans?

I can understand getting as far as VC pitches while overwhelmed with fixating on other aspects of the business problems, and still passing the "does this person have a good enough chance to have a big exit" gut feel test of the VCs. But are there no ongoing checks and advising, so that people don't miss everything else?

Re: We in-housed our data labelling

#8
post #6

I think this didn't age well, for HN, and it prompts some serious questions about our techbro startup culture. > Obvious but necessary: to incentivize productive work, we tie compensation to the number of characters transcribed, and assess financial penalties for failed tests (more on tests below). Penalties are priced such that subpar performance will result in little to no earnings for the labeller. So, these aren'…

How else would you do this?

Re: We in-housed our data labelling

#9
post #6

I think this didn't age well, for HN, and it prompts some serious questions about our techbro startup culture. > Obvious but necessary: to incentivize productive work, we tie compensation to the number of characters transcribed, and assess financial penalties for failed tests (more on tests below). Penalties are priced such that subpar performance will result in little to no earnings for the labeller. So, these aren'…

Anyone have the link to that thread?

Re: We in-housed our data labelling

#10
Where I'm from, "in house" means employees. I see "contractors" and "negative earnings" in the same article.

They do say that reviewers have to have some kind of aviation experience. I'd be more curious reading an article about how they source the talent here.

Post reply on HN