“Magic links” can end up in Bing search results, rendering them useless

Post by rsbadger » Mon, Jun 27, 2022, 12:33 PM UTC

Xylakant wrote: post #54

rsbadger wrote: post #17

Earlier quoted context omitted.

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

But there is nothing to indicate either in the post or in the referenced SO thread that the URLs are published to the search results. They are visited by bingbot, that much seems confirmed, but there’s no example where one of these results shows up in the public search results.

Post by hazmazlaz » Mon, Jun 27, 2022, 12:36 PM UTC

logifail wrote: post #69

Even if Microsoft claim this is about security scanning, isn't it fairly trivial to configure your webserver to serve up different content depending on the User-Agent request header? BingBot scans the link, gets a dummy page with 'clean' content, Microsoft delivers the email message to the user, user clicks through the link with actual browser, gets phishing / malware content...

Post by rsbadger » Mon, Jun 27, 2022, 12:41 PM UTC

cube00 wrote: post #63

rsbadger wrote: post #16

Earlier quoted context omitted.

It was blocked by robots.txt but Bing chose to ignore it. I even tried "blocking" the URLs in Bing webmaster tools today and this was the response: "Block request denied We found that the URL submitted for block is important for Bing users and hence cannot be blocked through Bing Webmaster Tools. We recommend that the best way to block URLs in this scenario is to add NOINDEX meta-tag to the HTML header of the page."

That is baffling logic. Sure, they think they know best and want to ignore the wishes of the owner of the web site. Why then respect a NOINDEX meta-tag instead of robots.txt?

Post by rsbadger » Mon, Jun 27, 2022, 12:42 PM UTC

dementiapatie wrote: post #57

FrenchDevRemote wrote: post #48

Earlier quoted context omitted.

> It might lead to sensitive data leak as cloud storage links can also be crawled to Bing no what might link to sensitive data leak is fools who store sensitive data on unprotected links

Agree, in my experience storage buckets are always private by default, and you must take several specific steps to make them public, ignoring the very big warnings sprinkled in each confirmation page along the way. Are there any cloud vendors that don't follow this approach?

Post by ratg13 » Mon, Jun 27, 2022, 12:50 PM UTC

nopassrecover wrote: post #65

ratg13 wrote: post #58

Earlier quoted context omitted.

Yes, but there is no indication they are publishing it in the search results. The original post is just complaining that the malware scanning is visiting the links. They come to the following conclusion > This effectively makes all one-time use links like login/pass-reset/etc useless. Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thin…

They are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)