Live data from Hacker News

Instructions around the usage of meta robot tags and robots.txt files

github.com

21–30 of 31 posts

Re: Instructions around the usage of meta robot tags and robots.txt files

#22

They should retroactively penalize hospitals for trying to hide this information as it is clearly against the spirit of the law.

The US Constitution prohibits “ex post facto” laws. You can punish someone for something that wasn’t a crime when they did it.

Re: Instructions around the usage of meta robot tags and robots.txt files

#25
post #8

Earlier quoted context omitted.

Why not both?

Because wouldn't they have to keep adding new techniques to the list continuously when in the other case they can say in what format the content needs to be available in.

Indexable is open to interpretation. No disallow directives in robots.txt isn't.

Re: Instructions around the usage of meta robot tags and robots.txt files

#26

They should retroactively penalize hospitals for trying to hide this information as it is clearly against the spirit of the law.

The US Constitution prohibits “ex post facto” laws. You can punish someone for something that wasn’t a crime when they did it.

Sure. I'm not saying make a new law then punish. I'm saying they should be punished based on the existing law that said prices had to be published. Similar to punishing someone for tax evasion even though the specific technique they used wasn't explicitly enumerated in the law.

Re: Instructions around the usage of meta robot tags and robots.txt files

#27
post #7

Earlier quoted context omitted.

Search engines can index pdfs.

Provided the content is not encrypted.

I stand corrected: encrypted PDFs are indexed unless a password is set. However, in the past they were unable to do so[0]: "Generally we can index textual content (written in any language) from PDF files that use various kinds of character encodings, provided they're not password protected or encrypted."

[0] https://developers.google.com/search/blog/2011/09/pdfs-in-go...

Re: Instructions around the usage of meta robot tags and robots.txt files

#28
As I read this, it seems like this is saying that if you are saying that you are posting public pricing information, you should not be hiding that information from search engines.

Seems like it's trying to avoid people (I guess it's targeted toward insurers, if I'm reading the main page correctly) saying that they are transparent about prices but hiding the information so no one sees it.

Please correct me if I'm wrong on this.

Re: Instructions around the usage of meta robot tags and robots.txt files

#29

Earlier quoted context omitted.

Provided the content is not encrypted.

I stand corrected: encrypted PDFs are indexed unless a password is set. However, in the past they were unable to do so[0]: "Generally we can index textual content (written in any language) from PDF files that use various kinds of character encodings, provided they're not password protected or encrypted." [0] https://developers.google.com/search/blog/2011/09/pdfs-in-go...

I think the assumption is that encrypted pdfs would not be used here because users still have to view these documents

Re: Instructions around the usage of meta robot tags and robots.txt files

#30
post #7
post #4

Earlier quoted context omitted.

And ban the use of PDF's - which is another way this could be avoided. Oh and mandate clean links no funky javascipt links that search crawlers don't follow.

Search engines can index pdfs.

Yeh but they don't work as well and wont be as discoverable
Post reply on HN