Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

171–180 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#171

If you're concerned about privacy, you shouldn't be using any Microsoft products, period.

true, but for the last 30ish years nobody cares about that opinion because it is profitable to use the smallest common denominator, get shit done and call it a day.

Re: “Magic links” can end up in Bing search results, rendering them useless

#172
post #40

tldr: In 2022, whether you are a paying customer or a free customer, YOU are the product and you will be squeezed for all you got (not specific to Microsoft at all)

Wow, you really think that all of the Fortune 500 companies using GSuite/Office365 are being squeezed for everything they have got?

Re: “Magic links” can end up in Bing search results, rendering them useless

#173
post #134

Earlier quoted context omitted.

They were indexed in Bing results, I’ve shared the URL of that in this thread

Holy shit thats bad. Do unlisted youtube and gdrive share links get indexed through this?

You’d assume those have proper robots.txt configuration?

Re: “Magic links” can end up in Bing search results, rendering them useless

#174
post #164

Earlier quoted context omitted.

They are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)

Worse would be links that are private to the people who posses the url. Like a private video on YouTube or a private document in google docs. The security depends on the URL being secret. This would silently publish secret information.

If those pages have no proper meta tags or robots.txt, there’s absolutely nothing wrong with this. Security by obscurity was never a good approach; from Proxies to security scanners, there has always been software that crawls unassuming URLs and published the results somewhere, if only a report to the admin.

Re: “Magic links” can end up in Bing search results, rendering them useless

#175

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

Wow! I think you just figured out an issue I had while working in a previous company using Auth0, where the authentication token would expire before the user had actually gone there (so the user saw an error page when clicking), but on our side it looked like the user went there but dropped off directly after. Had maybe 1% of the users complaining about this, but we never found the root-cause (we moved to our own authentication before we could figure it out). This has to have been why. Thanks for sharing this!

Re: “Magic links” can end up in Bing search results, rendering them useless

#176

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

HTTP GET requests are supposed to be idempotent, meaning that when you call an URL twice it should not lead to any different result compared to calling it once. This is part of the HTTP standard.

So while I think what Outlook does here is wrong, what these webpages do is simply a bug that should be fixed and shows a lack of understanding of HTTP.

Re: “Magic links” can end up in Bing search results, rendering them useless

#177
post #176

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

HTTP GET requests are supposed to be idempotent, meaning that when you call an URL twice it should not lead to any different result compared to calling it once. This is part of the HTTP standard. So while I think what Outlook does here is wrong, what these webpages do is simply a bug that should be fixed and shows a lack of understanding of HTTP.

> shows a lack of understanding of HTTP

I think that's a bit too much. Nothing in that suggests that they are breaking anything in the HTTP specification. You're right that GET requests has to be idempotent, but the exchange from the single-time use code you get in email with the API token, is most likely behind a non-GET request (like POST). The HTTP server responds to GET requests with the static assets (HTML/CSS/JS), but then the static assets has JavaScript that calls the POST endpoint for the exchange.

At least that's my guess. I agree it's a bug on their side, and they should fix it. But I think it's more of a UX issue than breaking the protocol.

Re: “Magic links” can end up in Bing search results, rendering them useless

#178

Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…

> Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug.

Alright, so imagine this: we have two endpoints GET "/page" and "POST /increment". Making a POST request to "/increment" increments a counter kept in memory and returns it's new value. The GET "/page" endpoint returns a HTML file, which contains JavaScript code that when executed, calls the "/increment" endpoint.

Are we now breaking the HTTP specification saying GET requests has to be idempotent if we visit "/page" in our browser? I think not, but this is sometimes how pages are implemented, which robots are gonna have to deal with, as otherwise many would consider it broken.

Don't get me wrong, I think it's a shitty implementation as well. But is it breaking the HTTP specification? Unlikely.

Re: “Magic links” can end up in Bing search results, rendering them useless

#179
post #119

Sounds like anyone dealing with any sort of vaguely sensitive information through email, and certainly any corporation, should avoid using Outlook for anything. The article is about email verification links, which is a pretty clear case where this can be dangerous, but tons of other links can get emailed without being intended for a wider audience. Besides, the fact that Outlook shares anything related to the content…

Or maybe you shouldn't rely on security through obscurity and instead should add a robots.txt as has been in the web standard since 1997.

I do have a robots.txt to block this directory. But Bing only listens to that for what to crawl, not what to index.

Re: “Magic links” can end up in Bing search results, rendering them useless

#180
post #125

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with…

Cool, so we should just stop building and running almost anything in existence because it's not all-encompassing? That sounds like a suboptimal path forward.
Post reply on HN