Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

Post reply on HN
Print view View original

11–20 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#11

Post by rsbadger »

matsemann wrote: post #4

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed. Same if you paste a link…

Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)

Re: “Magic links” can end up in Bing search results, rendering them useless

#12

Post by rsbadger »

tyingq wrote: post #5

>As of Feb 2017 Outlook ( https://outlook.live.com/ ) scans emails Makes me curious if only the free, online, Outlook does this. There's also paid O365 online Outlook and the fat client Outlook.

Exactly what I was thinking - what really worries me is it also seems to have happened to a lot of @gmail users too. I still can't figure out how Bing managed to find email tokens sent to gmail. Maybe users who connected their gmail account to Outlook...?

Re: “Magic links” can end up in Bing search results, rendering them useless

#14

Post by bigtones »

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

Re: “Magic links” can end up in Bing search results, rendering them useless

#15

Post by rsbadger »

swiftcoder wrote: post #9

Isn't this just the "link preview" feature, that is enabled by default in outlook? Many email clients generate link previews so that they can display a thumbnail of the webpage. Would seem to be necessary to filter out those referrers from the validate link

Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter.

Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.

Re: “Magic links” can end up in Bing search results, rendering them useless

#16

Post by rsbadger »

andix wrote: post #13

Can’t you block it with robots.txt or some similar method?

It was blocked by robots.txt but Bing chose to ignore it. I even tried "blocking" the URLs in Bing webmaster tools today and this was the response:

"Block request denied We found that the URL submitted for block is important for Bing users and hence cannot be blocked through Bing Webmaster Tools.

We recommend that the best way to block URLs in this scenario is to add NOINDEX meta-tag to the HTML header of the page."

Re: “Magic links” can end up in Bing search results, rendering them useless

#17

Post by rsbadger »

bigtones wrote: post #14

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

Re: “Magic links” can end up in Bing search results, rendering them useless

#18

Post by Semaphor »

matsemann wrote: post #4

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed. Same if you paste a link…

> AFAIK most email providers (like Gmail) will also visit links.

I keep hearing this, but our newsletter system has been using GET unsubscribe links since at least 2007 (but probably longer), and we never found a wave of Gmail users unsubscribing, we still have a lot of them. I wonder if this is simply an urban legend, if Gmail tries to recognize unsubscribe links, or if there is something else going on.

Re: “Magic links” can end up in Bing search results, rendering them useless

#19

Post by bigtones »

rsbadger wrote: post #11
matsemann wrote: post #4

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed. Same if you paste a link…

Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)

Can you prove that by linking to a Bing search where one of your pages show up ?

Re: “Magic links” can end up in Bing search results, rendering them useless

#20

Post by kenniskrag »

matsemann wrote: post #4

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed. Same if you paste a link…

yes but for login tokens the bing bot would be able to login none the less. Probably the login url would be in some logs at microsoft or antivirus vendor. It sounds paranoid but basically the url is a cleartext password laying around.
Post reply on HN