Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Larry Page on Real Time Google: We Have To Do It

readwriteweb.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Larry Page on Real Time Google: We Have To Do It

#11

Post by inerte »

buggy_code wrote: post #6
Alex3917 wrote: post #3

Please don't do this. Google is already useless enough as is. New HN comments get indexed in like five minutes, but then they're gone in a month or two. Same with all content lately, it gets indexed really fast and then it's gone. It's gotten to the point where I'm looking for stuff I know exists and I know the keywords used on the page, and I still can't find it because Google just doesn't index that page anymore ev…

Can you explain this more? Why would HN comments vanish? Doesn't google follow all links?

Personal anecdote, I have no idea if it's 100% related to OP:

http://news.ycombinator.com/item?id=576727

Now google today for "80-19-1"... it's not even on the second page (and it was the 3rd result to me too previously).

I guess OP's point is that Google is "happy" with fresh content, puts it on the front page, then the evil algorithms ;) push it farther.

Re: Larry Page on Real Time Google: We Have To Do It

#12

Post by aristus »

thisduck wrote: post #5

How does Google figure out relevance in realtime? With twitter, it's user driver content with tags etc. But with the net at large, blogs etc, this becomes difficult. Incoming links etc are hard to determine in real time (primarily because they haven't occurred yet).

The site's PR, uptime/age, update rate, uniqueness of content. By this measure HN is near the ideal: it has amazing inbound PR but does not link out much. It's been running fast and fine for ~3 years, the content is often unique, in the sense that it contains phrases Googlebot has never encountered before.

An experiment: here is a search that matches an exact phrase in this comment. http://www.google.com/search?q=%22By+this+measure+HN+is+near...

At the moment it returns nothing. within a minute or two this comment till be the first result.

Re: Larry Page on Real Time Google: We Have To Do It

#13

Post by Alex3917 »

buggy_code wrote: post #6
Alex3917 wrote: post #3

Please don't do this. Google is already useless enough as is. New HN comments get indexed in like five minutes, but then they're gone in a month or two. Same with all content lately, it gets indexed really fast and then it's gone. It's gotten to the point where I'm looking for stuff I know exists and I know the keywords used on the page, and I still can't find it because Google just doesn't index that page anymore ev…

Can you explain this more? Why would HN comments vanish? Doesn't google follow all links?

Actually today Google seems to be indexing most of the site. But usually when I do searches like:

site:news.ycombinator.com alex3917 "rule of thumb"

Only half of those posts show up, and to find the rest I need to use searchyc.com

Re: Larry Page on Real Time Google: We Have To Do It

#14

Post by ntoshev »

aristus wrote: post #9
buggy_code wrote: post #6

Earlier quoted context omitted.

Can you explain this more? Why would HN comments vanish? Doesn't google follow all links?

HN was not designed with SEO and linkage in mind -- after a while pages can become "orphaned" after they get pushed down by new stuff. No search engine follows all links. As a practical matter they generally give up after 6-8 links away from the home page.

Google search for this site still returns years old results:

http://www.google.bg/search?q=site:http://news.ycombinator.c...

(for better results try "800 days ago" with quotes, HN strips them for some reason)

Re: Larry Page on Real Time Google: We Have To Do It

#15

Post by foulmouthboy »

aristus wrote: post #12
thisduck wrote: post #5

How does Google figure out relevance in realtime? With twitter, it's user driver content with tags etc. But with the net at large, blogs etc, this becomes difficult. Incoming links etc are hard to determine in real time (primarily because they haven't occurred yet).

The site's PR, uptime/age, update rate, uniqueness of content. By this measure HN is near the ideal: it has amazing inbound PR but does not link out much. It's been running fast and fine for ~3 years, the content is often unique, in the sense that it contains phrases Googlebot has never encountered before. An experiment: here is a search that matches an exact phrase in this comment. http://www.google.com/search?q=%22…

52 minutes and counting. ;)

Re: Larry Page on Real Time Google: We Have To Do It

#16

Post by ajju »

I have a feeling Larry is being sneaky here and trying to misdirect competitors. I don't doubt that indexing content in real time and making it searchable has some utility - see the comment on this article about some question on the t.v. show Lost - but maybe 1/100 of my searches are like that. If my search for the missing link IDA is going to be polluted by hundreds of results of people tweeting about its discovery, I'd rather not have real time search.

On the other hand, this is not an either or proposition. I am ok with this as long as Google keeps the Tweet search results separate but equal (somewhat like they keep the blog search results separate via blog search but equal in that blog posts with good pageranks do appear in search results. Although I can imagine few if any individual tweets having a very high page ranks).

Re: Larry Page on Real Time Google: We Have To Do It

#17

Post by aristus »

foulmouthboy wrote: post #15
aristus wrote: post #12

Earlier quoted context omitted.

The site's PR, uptime/age, update rate, uniqueness of content. By this measure HN is near the ideal: it has amazing inbound PR but does not link out much. It's been running fast and fine for ~3 years, the content is often unique, in the sense that it contains phrases Googlebot has never encountered before. An experiment: here is a search that matches an exact phrase in this comment. http://www.google.com/search?q=%22…

52 minutes and counting. ;)

Yep. Fail. Stuff from yesterday is indexed however:

http://www.google.com/search?q=confusingly+called+copy-regio...

Re: Larry Page on Real Time Google: We Have To Do It

#18

Post by avichal »

ajju wrote: post #16

I have a feeling Larry is being sneaky here and trying to misdirect competitors. I don't doubt that indexing content in real time and making it searchable has some utility - see the comment on this article about some question on the t.v. show Lost - but maybe 1/100 of my searches are like that. If my search for the missing link IDA is going to be polluted by hundreds of results of people tweeting about its discovery,…

2 thoughts: 1) I worked on the search engine and Larry has been saying this for years. I worked on search quality back in 2005 and even back then he was talking about indexing everything and indexing it in seconds instead of hours.

2) Search is won on the margins. Yahoo and Google do equally well on most queries, but users decide which engine is better based on how it performs over all of the types of searches they have to do. So when you take the 80/100 searches you do that are not real time and use unique keywords and you know what you're looking for Y! and G come out the same. It's on those other 20/100 that Google wins users.

Re: Larry Page on Real Time Google: We Have To Do It

#20

Post by ajju »

avichal wrote: post #18
ajju wrote: post #16

I have a feeling Larry is being sneaky here and trying to misdirect competitors. I don't doubt that indexing content in real time and making it searchable has some utility - see the comment on this article about some question on the t.v. show Lost - but maybe 1/100 of my searches are like that. If my search for the missing link IDA is going to be polluted by hundreds of results of people tweeting about its discovery,…

2 thoughts: 1) I worked on the search engine and Larry has been saying this for years. I worked on search quality back in 2005 and even back then he was talking about indexing everything and indexing it in seconds instead of hours. 2) Search is won on the margins. Yahoo and Google do equally well on most queries, but users decide which engine is better based on how it performs over all of the types of searches they h…

Update: Hey, you're the guy who launched Google Transit! I could use your advice ( See http://www.ridecell.com/gt/about/ ). Can I email you?

I think indexing everything in seconds could definitely be a competitive advantage. I haven't tried Yahoo in years and back then it did much worse than Google. If it has improved this much, it makes sense that the competition is at the margins.

On the other hand, TechCrunch/Twitter et al's idea of real time search seems to be limited to indexing Twitter and Facebook updates as they happen. The arguments they present amount to "Someone tweeted about a plane crash from the crashed plane". I don't think indexing such tweets is going to be Google's edge in search. OTOH I can think of using such information to generate Google Alerts being very useful for some people.

Post reply on HN