I've noticed Office 365 Safe Links makes an OPTIONS request, not GET. So, restricting the endpoint to GET, via [HttpGet] decorator for example, may be a quick resolution.
“Magic links” can end up in Bing search results, rendering them useless
211–220 of 243 posts
Re: “Magic links” can end up in Bing search results, rendering them useless
#212Earlier quoted context omitted.
They are called magic links... only thing magic about them is their ability to annoy me
I think they exist to simplify the flow for the subset of users who end up using the Reset Password link each time their session expires. And I think that subset is much larger than some would expect.
This. You'd be amazed how many users just do a password reset each time to login instead of remembering their login info.
Re: “Magic links” can end up in Bing search results, rendering them useless
#213Earlier quoted context omitted.
They are called magic links... only thing magic about them is their ability to annoy me
>They are called magic links... only thing magic about them is their ability to annoy me I love them and prefer them to creating yet another account with a password.
Me too!
Re: “Magic links” can end up in Bing search results, rendering them useless
#214Earlier quoted context omitted.
It is common for corporate email security appliances as well. URLs should not be used for authentication neither should email. I really want to pick brains of people that work on these types of systems to see why they don't think so.
Is it ok to do a password reset through email? Because once you can do that you basically have email based authentication. The password only makes this autentication less secure and it's not needed.
It is not. You can initiate password reser via email but additional recovery controls like security questions should still be required. In an ideal world you have 2fa as well, if you reset that via email as well the it isn't actually 2fa, it is email based 1fa with extra steps. If your 2fa has a separate mechanism for recovery as well,that would be ideal. If it was my webapp, I would use hashes of 3 answers to user chosen questions, hashes in the browser/client. It could be an object pairing as well, every user gets a list of 30 objects or so and they pick 3 pairs as a recovery combination.
Re: “Magic links” can end up in Bing search results, rendering them useless
#215I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…
>We ended up adding a custom page to handle e-mail validation so we could handle the situation where the user lands on the page and the address has already been verified. That's a yikes from me! So I can sign up on your service as anyone with an Outlook account, without verification?
Discord let someone sign up with my gmail email address, sent an email verification link, and before I saw either the "welcome to Discord" or "please verify your email" links they'd already let the person in as me. I don't know if this is because of google crawling links from mail or some other kind of failure, but I wasn't pleased that Discord would let someone impersonate me.
Re: “Magic links” can end up in Bing search results, rendering them useless
#216Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…
> Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. Alright, so imagine this: we have two endpoints GET "/page" and "POST /increment". Making a POST request to "/increment" increments a counter kept in memory and returns it's new value. The GET "/page" endpoint returns a HTML file, which c…
From the spec, although it doesn't explicitly say 'dont automatically send a POST upon opening a GET', I think it's fairly clear its against the spirit of what a GET should represent to the user if it isn't a safe POST request.
"In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.
Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them."
Re: “Magic links” can end up in Bing search results, rendering them useless
#217Isn't this just the "link preview" feature, that is enabled by default in outlook? Many email clients generate link previews so that they can display a thumbnail of the webpage. Would seem to be necessary to filter out those referrers from the validate link
Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter. Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.
Is this actually the Bingbot crawling for the index, or do they just use the same bingbot code to generate link previews?
If I have a well-tested crawler sitting here, I don't see why I'd write a brand new one just to fetch previews in outlook...
Re: “Magic links” can end up in Bing search results, rendering them useless
#218Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…
That's a bit of a narrow view on this problem. When sending a link to someone, you expect that someone to view the link. Not some random mail service. Who gave the mail server permission to access the page? What if it contains copyrighted material? What if it's one of the millions of pages which don't follow the HTTP design philosophy to the letter? This is a can of worms.
When sending a link to someone, you expect your antivirus, your email provider, your email provider's spam filter, any intermediate email providers, the recipient email provider's spam filter, the recipient's email provider, the recipient's antivirus, your recipient's mail client, your recipient, and any other people who the message will be forwarded to, to see the link and evaluate it. The email standard is pretty clear that any number of intermediate servers and services can and will be able to see what you're sending.
Email isn't WhatsApp, there are probably at least three or four parties who will scan the link in any way they like. If you control your side you can make sure there are only one or two parties scanning the email en route, but the number can never be guaranteed to be zero without workarounds.
Who gave the mail server permission to access the page? The person who set up email on the domain. If you don't trust the hostmaster, don't send email to that domain. What if it contains copyrighted materials? Well, you just shared a plaintext link with a whole bunch of people, depending on your local legislation you may be in trouble.
You can't even expect a link clicked once by a single user in a browser to only appear once on the server side. TCP connections get dropped and retried. This isn't some kind of philosophical interpretation of a mystical protocol spec, this happens in real life. If you use POST/PUT/whatever requests, the user agent will prompt the user if they really want to repeat a request; this protection has been built in for years. It's just how browsers work and how they've been working for decades.
If your recipient is behind a proxy, the link may be visited several times each hour for up to a month while the proxy refreshes its cache. This was a more prevalent problem back in the day, these days web proxies are mostly a thing of the past; however, proxies still exist, and if you don't pay attention to those things they will bite you in the ass.
In real life bugs happen. That's fine in these cases, web dev isn't exactly rocket science, bugs are tolerated and can be fixed. The bug here isn't the fact that links get visited twice, though: the bug here is that the developers who set up their magical links forgot about idempotency when they wrote their code, or they chose to ignore the problem because they never ran into it themselves. Either way, the responsibility to get it fixed isn't on anyone but the party violating the spec.
As a workaround, S/MIME or PGP should work around most of these problems as intermediate servers can't see what's going on. What the client's machine will do with the decrypted message is still up to interpretation, of course.
Re: “Magic links” can end up in Bing search results, rendering them useless
#219Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…
> Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. Alright, so imagine this: we have two endpoints GET "/page" and "POST /increment". Making a POST request to "/increment" increments a counter kept in memory and returns it's new value. The GET "/page" endpoint returns a HTML file, which c…
I don't think it's breaking the spec per se, but web crawlers execute javascript and people hit reload on their browsers, sometimes accidentally. Automating this process may not be the solution here.
Personally, I think web crawlers like Bing shouldn't be executing javascript at all but front developers can't go without their client side rendering frameworks so search engines are more or less forced to.
As for a security mechanism, you want to emulate a browser as closely as possible to detect tricks like redirects from safe domains to attack domains and obfuscated URL crap. I'd expect any automated, non-interactive code to execute in a security analysis sandbox.
Is this breaking the standard? Who knows. What is a cloud antivirus but a web user agent running in a data center? The email protocol doesn't specify how the client should deal with links, the robots.txt only works for spiders, not for manually submitted URLs like those clicked in emails, and without a noindex tag you're going to see your page indexed by the mail provider company regardless of what your robots file says.
I think in theory your solution solves the spec breaking problem, but it doesn't solve the problem in practice because there are many other components for which there are no standards and defensive programming is required.
Re: “Magic links” can end up in Bing search results, rendering them useless
#220Earlier quoted context omitted.
Anyone paying for the firehose access can do this.
That's true of course. What's interesting to me is that they've decided to pay for this access and visit the links so quickly. It must be pretty expensive or hard to get if only around two-dozen companies pay for access to the data[0]. [0] https://www.washingtonpost.com/technology/2022/06/08/elon-mu...
It might be that Apple is paying for the firehose as a data-source to bootstrap its search engine. Don't they have one accessible via Siri already ? (I don't follow Apple tech very closely).