21–30 of 243 posts
There's a difference between adding the URLs to search engine results and accessing the URLs to scan for malware. The latter is quite common, lots of email hosts do that. It's not clear to me from the post if the former is actually happening - the author doesn't state that they found the links in Bing's results, just that they were accessed by BingBot.
I found them in Bing results
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.
From https://developers.google.com/search/docs/advanced/robots/in...
Earlier quoted context omitted.
Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)
Can you prove that by linking to a Bing search where one of your pages show up ?
Whenever I buy a flight, google puts the date on "my" calendar.
Just lets not pretend Microfsoft is especially bad at this, ok?
Isn't this just the "link preview" feature, that is enabled by default in outlook? Many email clients generate link previews so that they can display a thumbnail of the webpage. Would seem to be necessary to filter out those referrers from the validate link
Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter. Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.
There's a difference between adding the URLs to search engine results and accessing the URLs to scan for malware. The latter is quite common, lots of email hosts do that. It's not clear to me from the post if the former is actually happening - the author doesn't state that they found the links in Bing's results, just that they were accessed by BingBot.
I found them in Bing results
Earlier quoted context omitted.
I found them in Bing results
Are the URLs being served with a “noindex” header? Blocking crawls with robots.txt cannot de-list items from Google or other search engines. > Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use anoth…