Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

181–190 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#181
post #176

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

HTTP GET requests are supposed to be idempotent, meaning that when you call an URL twice it should not lead to any different result compared to calling it once. This is part of the HTTP standard. So while I think what Outlook does here is wrong, what these webpages do is simply a bug that should be fixed and shows a lack of understanding of HTTP.

It's idempotent. Your account won't get un-verified if you open the verification link twice.

Idempotent != side-effect free

Re: “Magic links” can end up in Bing search results, rendering them useless

#182

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

GET requests are supposed to be idempotent

Re: “Magic links” can end up in Bing search results, rendering them useless

#183
post #125

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with…

I believe the recommended practice is to hover over the URL before clicking the link.

If you do so, in Outlook, there will be a pop that shows "Original URL: XXX". This allows users to make a determination for themselves whether the link is safe or not.

Re: “Magic links” can end up in Bing search results, rendering them useless

#184

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

If this were security scanning, why does it identify itself as BingBot? Doesn't that just allow cloaking and offer an easy workaround for any adversary with a modicum of intelligence?

Re: “Magic links” can end up in Bing search results, rendering them useless

#185
post #104

Earlier quoted context omitted.

It is common for corporate email security appliances as well. URLs should not be used for authentication neither should email. I really want to pick brains of people that work on these types of systems to see why they don't think so.

Many people (most?) prefer to signup to services by email address. To do so, those email addresses must be verified. How would you verify it without sending them an email link?

options include:

* use an interstitial page so that the actual activation is a POST request;

* send a confirmation code instead of a link

Re: “Magic links” can end up in Bing search results, rendering them useless

#186

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

Does Gmail do this?

They definitely do link re-writing. As to what use they make of the original href attributes, I don't know.

Re: “Magic links” can end up in Bing search results, rendering them useless

#187
post #125

Earlier quoted context omitted.

> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with…

I believe the recommended practice is to hover over the URL before clicking the link. If you do so, in Outlook, there will be a pop that shows "Original URL: XXX". This allows users to make a determination for themselves whether the link is safe or not.

We got some security courses about that too. Unfortunately, outlook replaces all of them with some safelink url rewriting, so the only way left to find out if a link is scammy is clicking it.

Re: “Magic links” can end up in Bing search results, rendering them useless

#188

Earlier quoted context omitted.

Agree, in my experience storage buckets are always private by default, and you must take several specific steps to make them public, ignoring the very big warnings sprinkled in each confirmation page along the way. Are there any cloud vendors that don't follow this approach?

Almost all of them. A good example is Dropbox link you send to someone. I could generate this link to a private file in my Dropbox, email it you, and Bing (may) index it. https://www.dropbox.com/s/vucien2ns8jktga/denim%20bodywarmer... I doubt many people realise this when they email "private" links...

[deleted]

Re: “Magic links” can end up in Bing search results, rendering them useless

#189
post #80

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

Wouldn't it be trivial to keep the list of malicious pages locally and not send any data?

My first guess is that giving phishers/scammers the list of all malicious domains/pages might allow them to circumvent it.

Re: “Magic links” can end up in Bing search results, rendering them useless

#190
post #176

Earlier quoted context omitted.

HTTP GET requests are supposed to be idempotent, meaning that when you call an URL twice it should not lead to any different result compared to calling it once. This is part of the HTTP standard. So while I think what Outlook does here is wrong, what these webpages do is simply a bug that should be fixed and shows a lack of understanding of HTTP.

> shows a lack of understanding of HTTP I think that's a bit too much. Nothing in that suggests that they are breaking anything in the HTTP specification. You're right that GET requests has to be idempotent, but the exchange from the single-time use code you get in email with the API token, is most likely behind a non-GET request (like POST). The HTTP server responds to GET requests with the static assets (HTML/CSS/J…

Agree, the commentor is hung up on their demonstrably superior understanding of HTTP.

We found at least one scanning service to be fetching the URL with the user agent of a browser, and executing JavaScript on the page.

A lot of our user interactions could be simpler, but this sort of behaviour led to many things being put behind a "go" button.

I only wonder how long before scanners and search engines start clicking these buttons to activate content on the page so they can scan/index it.

Post reply on HN