Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Four lessons from a year building tools for machine learning

humanloop.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Four lessons from a year building tools for machine learning

#11

Post by RicoElectrico »

razcle wrote: post #10
ska wrote: post #9

I find #1 "Subject matter experts have as much impact as data scientists" surprising only in that it was considered surprising.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

Compare this with the famous quote:

> Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek

Re: Four lessons from a year building tools for machine learning

#12

Post by ska »

razcle wrote: post #10
ska wrote: post #9

I find #1 "Subject matter experts have as much impact as data scientists" surprising only in that it was considered surprising.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

> its definitely over looked in much of academic ML

right - but it seems like a, if not the, first lesson you learn when you leave the classroom for the "real" world.

It's not surprising to me that any people think this way, but it seems to be a characteristic of inexperience (or narrow experience).

Re: Four lessons from a year building tools for machine learning

#13

Post by razcle »

RicoElectrico wrote: post #11
razcle wrote: post #10

Earlier quoted context omitted.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

Compare this with the famous quote: > Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek

I very nearly said this myself!

I think the mistake of this quote is in the application of the expertise. The bitter lesson is that data + compute can outperform inductive biases but that doesn't mean you don't need domain expertise to get the right data.

Re: Four lessons from a year building tools for machine learning

#14

Post by razcle »

ska wrote: post #12
razcle wrote: post #10

Earlier quoted context omitted.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

> its definitely over looked in much of academic ML right - but it seems like a, if not the, first lesson you learn when you leave the classroom for the "real" world. It's not surprising to me that any people think this way, but it seems to be a characteristic of inexperience (or narrow experience).

Maybe so but most data science workflows still don't acknowledge this "obvious" truth.

Re: Four lessons from a year building tools for machine learning

#15

Post by ska »

RicoElectrico wrote: post #11
razcle wrote: post #10

Earlier quoted context omitted.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

Compare this with the famous quote: > Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek

> Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek

This one is easy to misapply. If you are applying your domain experts to the model, you might have a bad time. If you are applying them to the data, most likely not. And data is usually more important than the model.

Re: Four lessons from a year building tools for machine learning

#16

Post by Imnimo »

>In just a few hours the lawyers had trained a model that provided the outcome of all 80,000 judgements without needing the input of a data scientist at all.

If this is meant to imply it predicted them all correctly, that rings alarm bells to me. 100% accuracy is much more likely to mean something is wrong (label leakage?) than it is to mean you have an amazing model.

Re: Four lessons from a year building tools for machine learning

#17

Post by [deleted] »

razcle wrote: post #14
ska wrote: post #12

Earlier quoted context omitted.

> its definitely over looked in much of academic ML right - but it seems like a, if not the, first lesson you learn when you leave the classroom for the "real" world. It's not surprising to me that any people think this way, but it seems to be a characteristic of inexperience (or narrow experience).

Maybe so but most data science workflows still don't acknowledge this "obvious" truth.

[deleted]

Re: Four lessons from a year building tools for machine learning

#18

Post by razcle »

Imnimo wrote: post #16

>In just a few hours the lawyers had trained a model that provided the outcome of all 80,000 judgements without needing the input of a data scientist at all. If this is meant to imply it predicted them all correctly, that rings alarm bells to me. 100% accuracy is much more likely to mean something is wrong (label leakage?) than it is to mean you have an amazing model.

No, not 100% accuracy. I've left out details for the sake of brevity but with a precision and recall high enough for the team to be able to answer the questions they cared about.

Re: Four lessons from a year building tools for machine learning

#19

Post by gumby »

razcle wrote: post #10
ska wrote: post #9

I find #1 "Subject matter experts have as much impact as data scientists" surprising only in that it was considered surprising.

I think this is one of those points that is obvious in retrospect but almost universally under appreciated. Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks. Perhaps it should be obvious but its definitely over looked in much of academic…

The 80s called and and want their subject matter training back

Re: Four lessons from a year building tools for machine learning

#20

Post by mpfundstein »

ska wrote: post #15
RicoElectrico wrote: post #11

Earlier quoted context omitted.

Compare this with the famous quote: > Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek

> Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek This one is easy to misapply. If you are applying your domain experts to the model, you might have a bad time. If you are applying them to the data, most likely not. And data is usually more important than the model.

> And data is usually more important than the model.

idk. we went from conv nets to transformers just to have the quality of our predictions go up as well as reducing the amount of data prep time by a factor of 20.

no change in data, just a better model.

in my field, improvements are nearly always made in the model. never in the data or data prep. (crowd countinf, people tracking, etc)

Post reply on HN