Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google bot now appears to emulate users interacting with the site

swapped.cc

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Google bot now appears to emulate users interacting with the site

#21

Post by Matt_Cutts »

LocalPCGuy wrote: post #3
drivebyacct2 wrote: post #2

It's been known for some time that Google tries to interact with content that it thinks is dynamic or retrieved via AJAX. For obvious reasons.

Interact, yes. Execute AJAX? Not so much. Matt Cutts said in April: "So Asynchronous Javascript is a little bit more complicated, and that’s maybe further down the road, but the common case is Javascript." From the short article, it seems like this is going a bit further than what Cutts is saying GoogleBot is capable.

That specific video came out in April, but we actually taped the video a few months earlier. Google continues to work on better/smarter ways to crawl websites, including getting better at executing JavaScript to discover content.

Re: Google bot now appears to emulate users interacting with the site

#23

Post by phpnode »

dsl wrote: post #15

I suspect Googlebot may be replaying a sequence of requests recorded with the Google Toolbar.

I think the user specific token in the URL disproves that. It's more likely that they're just doing the discovery themselves. Otherwise googlebot would be responsible for massive data loss, as it goes around mistakenly replaying delete requests on behalf of toolbar users...

Re: Google bot now appears to emulate users interacting with the site

#24

Post by tintin »

yaix wrote: post #22

GBot has been executing Javascript and hitting URLs the Javascript generates, for quite some time. And very likely it also evaluates the page layout as it is influences by Javascript, to detect keyword stuffing hidden by Javascript.

They also started looking at content above the fold and also monitor request time vs page loaded time for a while. I think it's great that Google is starting to monitor the user experience. It will keep people from building Javascript bloated sites that don't respond well.

I hope this isn't becoming a trend, but lately I see a lot of responsive sites that don't respond to user input. Maybe a little force from Google will stop this trend.

Re: Google bot now appears to emulate users interacting with the site

#25

Post by yaix »

tintin wrote: post #24
yaix wrote: post #22

GBot has been executing Javascript and hitting URLs the Javascript generates, for quite some time. And very likely it also evaluates the page layout as it is influences by Javascript, to detect keyword stuffing hidden by Javascript.

They also started looking at content above the fold and also monitor request time vs page loaded time for a while. I think it's great that Google is starting to monitor the user experience. It will keep people from building Javascript bloated sites that don't respond well. I hope this isn't becoming a trend, but lately I see a lot of responsive sites that don't respond to user input. Maybe a little force from Google…

The new Goog search interface being one of them, especially on low end computers (netbooks etc).

The results of Goog monitoring page load and render times can be seen in Goog's Webmaster Tools and is measured by user's browsers. They started doing that some years ago and I am pretty sure by now it is part of the ranking.

And at the same time they made their own site slower and bloaded. :-/

Re: Google bot now appears to emulate users interacting with the site

#26

Post by VMG »

tintin wrote: post #24
yaix wrote: post #22

GBot has been executing Javascript and hitting URLs the Javascript generates, for quite some time. And very likely it also evaluates the page layout as it is influences by Javascript, to detect keyword stuffing hidden by Javascript.

They also started looking at content above the fold and also monitor request time vs page loaded time for a while. I think it's great that Google is starting to monitor the user experience. It will keep people from building Javascript bloated sites that don't respond well. I hope this isn't becoming a trend, but lately I see a lot of responsive sites that don't respond to user input. Maybe a little force from Google…

> It will keep people from building Javascript bloated sites that don't respond well.

On the other hand it will finally allow sites that do client-side rendering with JavaScript to be indexed properly, provided that they are responsive - which isn't that hard to do.

Re: Google bot now appears to emulate users interacting with the site

#28

Post by Produce »

Produce wrote: post #13
nicksergeant wrote: post #11

Earlier quoted context omitted.

This isn't Reddit.

Cry me a river.

On a more serious note, a system which would probably solve this problem is to let users tag comments with one of a set of categories. E.g. the slashdot system (funny, insightful, etc) but user-initiated. Combine that with per-user settings on what type of comments they want to see and you could simply hide any jokes or less relevant information.

Re: Google bot now appears to emulate users interacting with the site

#29

Post by idleloops »

yaix wrote: post #22

GBot has been executing Javascript and hitting URLs the Javascript generates, for quite some time. And very likely it also evaluates the page layout as it is influences by Javascript, to detect keyword stuffing hidden by Javascript.

I can confirm this as a site I worked on utilised JS + CSS for the theme (deliberate choice I might add,) pages preview correctly in search results.

Re: Google bot now appears to emulate users interacting with the site

#30

Post by KaoruAoiShiho »

latitude wrote: post #12
KaoruAoiShiho wrote: post #10

I have a dynamic site that I'm pretty sure google does not run the javascript. How do I get google to run the javascript?

Do you implement _escaped_fragment_ semantics?

I do not. I use pushstate instead of hashbang.
Post reply on HN