Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wouldn't it be fun to build your own Google?

radar.oreilly.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Wouldn't it be fun to build your own Google?

#41

Post by pjbrunet »

At first Google was a search algorithm, but at some point they decided to have humans review and rank the important queries. Important as in query volume.

Why use humans? People can decide if your navigation is intuitive. They can decide if your page looks like crap. If 230,000 people are searching for "coconut oil" per month (actual numbers) then it's worth having an intern spend 15 minutes to make sure page 1 of "coconut oil" looks right.

Google can afford that. They need a human to decide if the "user experience" is actually good vs. disallowing the back button and forcing the browser to crash, which is how I suppose you could fake a "time on site" metric if this was just an algorithmic problem.

Google is now more like playing Zork. You type "Go North" like 10 million other people before you typed "Go North" and Google has already crafted that experience you'll find in next room. (Which makes me wonder, do they score how boring you are based on predictability?) This is becoming more and more obvious over time as a search for "calculator" shows you an actual calculator that a human at Google created. That's not an algorithmic response.

Similarly, I see that human touch coming more into play with voice recognition, Google Glass, Siri, etc. Call that "AI" or whatever. You ask Google a question and Google has already sculpted a slick answer based on tons of testing. That's how I see Google as a search engine now. Part of the crawling is interesting (recognizing objects in photos?) but I think human reviews of all the important websites and SERPs, that's harder for a competitor to reproduce.

Re: Wouldn't it be fun to build your own Google?

#42

Post by ahpeeyem »

jobposter1234 wrote: post #40
threeseed wrote: post #37

Earlier quoted context omitted.

Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place. I understand Google is trying to be clever here and appealing to novices who don't really understand what they want. But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find sema…

>Isn't my issue obvious ? Not touching that with a 10 foot pole... (just teasing, just teasing...) But seriously, it sounds like what you want is a keyword matching engine. Google, for better or worse, has decided they know enough about their user's searches that they don't mind modifying the query parameter in an attempt to retrieve what people want rather than what they literally say they want. I understand that yo…

After Google Plus was released, Google changed its search syntax slightly so that you now need to "quote" mandatory terms instead of prefixing them with a + symbol.

http://waxy.org/2011/10/google_kills_its_other_plus/ https://productforums.google.com/forum/#!topic/websearch/3oI...

Re: Wouldn't it be fun to build your own Google?

#43

Post by stevesalevan »

Smerity wrote: post #10

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…

So awesome to see that folks are discovering Common Crawl, it's a really great project and the amount of data they've crawled has been tremendous. If you're looking to get into machine learning, it's perhaps the most comprehensive dataset you can get your hands on.

Re: Wouldn't it be fun to build your own Google?

#45

Post by nostrademons »

threeseed wrote: post #37
jobposter1234 wrote: post #35

Earlier quoted context omitted.

Isn't google doing that because it detected the semantic information was on the page, even if the exact term wasn't? Is your issue with the fact that they're doing more than just a keyword retrieval, or is your issue with the fact that they're doing it poorly?

Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place. I understand Google is trying to be clever here and appealing to novices who don't really understand what they want. But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find sema…

There's a feature for that: Verbatim mode.

https://www.google.com/search?q=webhcat+primary+key&tbs=li:1

Re: Wouldn't it be fun to build your own Google?

#47

Post by Animats »

pjbrunet wrote: post #41

At first Google was a search algorithm, but at some point they decided to have humans review and rank the important queries. Important as in query volume. Why use humans? People can decide if your navigation is intuitive. They can decide if your page looks like crap. If 230,000 people are searching for "coconut oil" per month (actual numbers) then it's worth having an intern spend 15 minutes to make sure page 1 of "c…

I think human reviews of all the important websites and SERPs, that's harder for a competitor to reproduce.

Google was forced into that by improved "search engine optimization". SEO used to be about things like keyword stuffing, but as Google made their search engine smarter, SEO companies made their search spamming smarter. There are now SEO operations using machine learning to reverse engineer Google's algorithms and then automatically spam just enough to stay under the threshold.

In 2010, Google tried using "local" data to improve search. That turned out to be extremely easy to spam. A classic example of this can be found by searching for "laptop repair bradford pa". This brings up "Illusory Laptop Repair", located in the middle of a railroad crossing. A SEO expert created that phony business listing to demonstrate how bad Google was at detecting such spam. Google still thinks it's real.

In 2012, Google tried using "social" data to improve search. That worked even worse. Fake Google accounts created to create fake "+1"s may have exceeded the number of real ones. Google "+1" are still for sale; the going rate is about $0.10 each.

Meanwhile, links aren't as useful as they used to be. Who creates a link to a retail outlet other than on social media any more? Google is trying all sorts of "signals", but in heavily spammed areas, they're not doing all that well.

Yandex has been trying search that doesn't weight links at all for some heavily spammed categories in the Moscow area. It seems to be working for fake real estate ads.

(We have a partial solution - find the real-world business behind the web site and check it out in hard data sources, such as Dun and Bradstreet or Experian, which have business credit data. See "sitetruth.com/doc".)

Re: Wouldn't it be fun to build your own Google?

#48

Post by rjaco31 »

Animats wrote: post #47
pjbrunet wrote: post #41

At first Google was a search algorithm, but at some point they decided to have humans review and rank the important queries. Important as in query volume. Why use humans? People can decide if your navigation is intuitive. They can decide if your page looks like crap. If 230,000 people are searching for "coconut oil" per month (actual numbers) then it's worth having an intern spend 15 minutes to make sure page 1 of "c…

I think human reviews of all the important websites and SERPs, that's harder for a competitor to reproduce. Google was forced into that by improved "search engine optimization". SEO used to be about things like keyword stuffing, but as Google made their search engine smarter, SEO companies made their search spamming smarter. There are now SEO operations using machine learning to reverse engineer Google's algorithms a…

>A classic example of this can be found by searching for "laptop repair bradford pa". This brings up "Illusory Laptop Repair", located in the middle of a railroad crossing. A SEO expert created that phony business listing to demonstrate how bad Google was at detecting such spam. Google still thinks it's real.

This doesn't seem to work.

Re: Wouldn't it be fun to build your own Google?

#49

Post by sparkzilla »

minthd wrote: post #38
sparkzilla wrote: post #28

The problem with algorithmic/scraper search methods, is that they only work with existing data. For example, most Google searches gives a list of websites on one side, and some data scraped from Wikipedia on the other. There is not much meaning there. That's because Google's algorithm cannot combine the results into something original, because that would require human creativity. As such, I see the rise of different…

You really discount the value of the long tail of search, which is where you get best info from google.

You may get the best (more precise) info in the long tail, but I'm pretty sure that Google makes most of its money from the most popular searches.

Re: Wouldn't it be fun to build your own Google?

#50

Post by desdiv »

jobposter1234 wrote: post #40
threeseed wrote: post #37

Earlier quoted context omitted.

Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place. I understand Google is trying to be clever here and appealing to novices who don't really understand what they want. But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find sema…

>Isn't my issue obvious ? Not touching that with a 10 foot pole... (just teasing, just teasing...) But seriously, it sounds like what you want is a keyword matching engine. Google, for better or worse, has decided they know enough about their user's searches that they don't mind modifying the query parameter in an attempt to retrieve what people want rather than what they literally say they want. I understand that yo…

I think, however, that you can preceded mandatory terms with a + sign to require it to be present on page.

That was my first reaction too, but I tried it out and it still doesn't work. In fact, the result is kinda funny:

Googling "webhcat primary key" got me 3 results with "webhcat" on the first page.

Googling "+webhcat primary key" got me 5 results in total, none of which contains the word "webhcat".

Post reply on HN