Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Rater' is paid $10 hourly to teach Google's algorithm

finance.yahoo.com

Post reply on HN
Print view View original

41–50 of 117 posts

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#41

Post by EugeneOZ »

fshbbdssbbgdd wrote: post #33
civilized wrote: post #23

Dumb question - why can't I just upvote and downvote pages, and Google enhances its ranking for me based on that? Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.

Google already has similar signals: 1. Do you click on the link in the results page? 2. After clicking on that link, do you go back to the search or not? 3. How long do you stay there before coming back? Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and fou…

It obviously doesn't work, otherwise stackoverflow ad-filled copies wouldn't exist.

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#42

Post by echen »

fshbbdssbbgdd wrote: post #33
civilized wrote: post #23

Dumb question - why can't I just upvote and downvote pages, and Google enhances its ranking for me based on that? Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.

Google already has similar signals: 1. Do you click on the link in the results page? 2. After clicking on that link, do you go back to the search or not? 3. How long do you stay there before coming back? Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and fou…

Note that it's often difficult to tell if these signals are good or bad. For example:

1. Clicking on a link is often a negative signal. If you're searching for "when was barack obama born?", hopefully you get the answer on the search results page itself, so you never have to click. Or, the caption for each search result should have a snippet containing the answer.

2. If you stay a long time on the search result you click on, is it because the information is hard to find or because the page is deep and useful?

I used to work on Search measurement at YouTube and Facebook, and run a human evaluation platform now. Here's an example what they can look like when you use high-quality raters: https://demos.surgehq.ai/google-vs-bing/

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#43

Post by scoopertrooper »

bryanrasmussen wrote: post #37
jancsika wrote: post #29

Earlier quoted context omitted.

If Google can't use it's vast array of user data to tell the difference between SEO bots and non-bot users, what's even the point of it collecting all that fucking data in the first place?

Good point, but I have my reasons for suspecting they can't tell the difference https://news.ycombinator.com/item?id=31296953

Maybe that's part of how they train it?

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#45

Post by civilized »

lordnacho wrote: post #24
civilized wrote: post #23

Dumb question - why can't I just upvote and downvote pages, and Google enhances its ranking for me based on that? Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.

Who do you reckon would cast the most votes? Correct, SEO bots.

Collaborative filtering: not just for movie recommendations anymore?

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#46

Post by omoikane »

ocdtrekkie wrote: post #13
rcktmrtn wrote: post #5

On a related note, am I right in understand that the "Google Experts" [1] do it for free? [1] https://developers.google.com/community/experts/

Yes, Google has tricked a massive army of people into doing free work for one of the most profitable companies in the world. Things like "Google Local Guides" give people a gold star (and maybe a pair of branded socks) for doing unpaid labor. It's kinda hilarious, but also sad. I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who…

I can see how the gamification aspect of points and levels might be more motivating for some people, but mostly I see Local Guides as not too different from people posting pictures and reviews to various social networks, except that it is more discoverable by people using Maps. I submit contents to Maps for mostly the same reasons why I use social networks and expect nothing in return.

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#47

Post by echen »

Proven wrote: post #4

How much should they be paid? It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job. And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h. As one comment says, it probably beats fliping burgers.

It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own).

For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be?

"neovim worktrees"

"django models through"

"pandas econdb"

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#48

Post by echen »

ggm wrote: post #44

Surprisingly badly paid. I wonder how strongly quality would track pay? Probably they act as a CAPTCHA mutually cross checking and so QA is by volume, and not coincidentally demands they work in isolation. Sad story.

Quality control is often indeed by volume. When I was at Google / Facebook, search evaluations would typically have 5 raters each. One of the difficulties, though, is that search evaluation can be quite subjective. And honestly, most of the search engine raters we used were low quality and did a bad job (so the aggregate answer was often nonsense).

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#49

Post by acheron »

vba616 wrote: post #18
ocdtrekkie wrote: post #13

Earlier quoted context omitted.

Yes, Google has tricked a massive army of people into doing free work for one of the most profitable companies in the world. Things like "Google Local Guides" give people a gold star (and maybe a pair of branded socks) for doing unpaid labor. It's kinda hilarious, but also sad. I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who…

>laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something. So when you go to a store that Google Maps says is open, and find it's closed, you refuse to correct the hours and go "MUHAHAHAHA...I refuse to help Google, take that , corporate scum!"

What? Why on earth would I correct it?

Re: 'Rater' is paid $10 hourly to teach Google's algorithm

#50

Post by bmacho »

EugeneOZ wrote: post #41
fshbbdssbbgdd wrote: post #33

Earlier quoted context omitted.

Google already has similar signals: 1. Do you click on the link in the results page? 2. After clicking on that link, do you go back to the search or not? 3. How long do you stay there before coming back? Adding an explicit up/downvote would capture some additional information, but it would overlap with the above, and most users would ignore the button. I’d guess Google has at some point experimented with this and fou…

It obviously doesn't work, otherwise stackoverflow ad-filled copies wouldn't exist.

Can you give me some?

I love stackoverflow content, but hate the surface, e.g. how it loads jquery from google (???) and if I disable it, then the content breaks, because obviously the only way to implement a toggleable div is when you send your users browsing history to google directly..

Post reply on HN