Instructions around the usage of meta robot tags and robots.txt files
1–10 of 31 posts
Re: Instructions around the usage of meta robot tags and robots.txt files
#2At least they are keeping up with current technical tricks.
Re: Instructions around the usage of meta robot tags and robots.txt files
#3Nice. They should probably add HTTP headers (X-Robots-Tag) to the list. Cloaking, too.
Re: Instructions around the usage of meta robot tags and robots.txt files
#4Nice. They should probably add HTTP headers (X-Robots-Tag) to the list. Cloaking, too.
And ban the use of PDF's - which is another way this could be avoided.
Oh and mandate clean links no funky javascipt links that search crawlers don't follow.
Re: Instructions around the usage of meta robot tags and robots.txt files
#5Why are they banning the methods instead of saying how the pricing should be displayed. E.g. why say you cannot have "noindex" instead of saying "the pricing information needs to be in plain-text, human-readable, accessible, indexable..." and so on.
Re: Instructions around the usage of meta robot tags and robots.txt files
#6Why are they banning the methods instead of saying how the pricing should be displayed. E.g. why say you cannot have "noindex" instead of saying "the pricing information needs to be in plain-text, human-readable, accessible, indexable..." and so on.
Why not both?
Re: Instructions around the usage of meta robot tags and robots.txt files
#7Re: Instructions around the usage of meta robot tags and robots.txt files
#8Why are they banning the methods instead of saying how the pricing should be displayed. E.g. why say you cannot have "noindex" instead of saying "the pricing information needs to be in plain-text, human-readable, accessible, indexable..." and so on.
Why not both?
Because wouldn't they have to keep adding new techniques to the list continuously when in the other case they can say in what format the content needs to be available in.