Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

Post reply on HN
Print view View original

231–240 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#231

Post by petercooper »

noisem4ker wrote: post #100
usrn wrote: post #70

Earlier quoted context omitted.

I just love it when they "scan" password reset links.

The HTTP GET method is idempotent by specification. Visiting a webpage should not trigger password resets or any other actions by itself. If that's a problem then it's the site's fault for being defective.

So how do you implement a "one click unsubscribe" link in an email? They're on GET requests. You could use JavaScript on the resulting page to then trigger the unsubscribe but bots are now running JavaScript as well.

Re: “Magic links” can end up in Bing search results, rendering them useless

#232

Post by hansvm »

phendrenad2 wrote: post #163
sofixa wrote: post #125

Earlier quoted context omitted.

> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with…

The problem with that is that the logic is broken. If 99.99% of phishing can be prevented this way, what problem do you have with it? Would you really catch that 0.01% that an automated system wouldn't?

You mean you don't verify calls to action via other information channels? Fairly regularly I get phishing emails that correctly spoof the crypto headers of major sites (e.g., because of a misconfigured mail service). If an email asks me to do something, it either doesn't get done or I cover my ass in as many ways as possible, no exceptions.

That isn't by itself an argument against a good automated system -- I definitely like not having to sift through most of that garbage, but catching the 0.01% should be a routine practice, not something that seems like an insurmountable burden.

Re: “Magic links” can end up in Bing search results, rendering them useless

#233

Post by withinboredom »

dmw_ng wrote: post #102
sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

Links like this are stupid regardless of Outlook's behaviour because they require a perfectly reliable client and network and user in a perfectly undisturbed flow. If I can't F5, if I double-click, if my mouse is wonky, my wifi is bad, my power goes out, my computer hangs, my DSL dies just after a click, if I accidentally close the tab.. there are any of a thousand reasons why abusing GET for a one-time-use page or r…

Haha, I was in a restaurant and you paid through your phone. My browser updated so it closed between the thank you page and the payment click. State was lost so the thank you page was broken. The restaurant didn’t think I paid but my bank account said otherwise (this was a bank transfer via ideal, not credit card). Getting out of there without paying twice was entertaining.

Re: “Magic links” can end up in Bing search results, rendering them useless

#234

Post by liam_ja »

rbut wrote: post #52

I have observed this, but also found that BingBot modifies the query string parameters of your URL. It does this by changing a character of the URL, possibly in an attempt to find new pages? I noticed this because I generate links with a signed token to ensure integrity and started receving invalid token crash reports in Sentry, always from BingBot.. To fix this I had to move the tokens from the query string into the…

I've finally found someone else who's seen this behaviour!

I've noticed this too, and I found (in my case anyway) that Bing/Outlook seems to Rot13 the keys of the query parameters - is this what you're seeing too?

Re: “Magic links” can end up in Bing search results, rendering them useless

#235

Post by yencabulator »

For what it's worth, here's Google admitting that GoogleBot causes POSTs:

https://developers.google.com/search/blog/2011/11/get-post-a...

Automatically triggered POST is not sufficient to keep the bots at bay.

They seem to be implying that only automatically triggered POST are acceptable, but that was also >10 years ago.

With the way things are going, it might be that any on-page confirmation buttons won't be sufficient to keep the bots at bay. Maybe it's time to fight back, check the user-agent, and serve the bots a CAPTCHA?

Re: “Magic links” can end up in Bing search results, rendering them useless

#236

Post by noisem4ker »

petercooper wrote: post #231
noisem4ker wrote: post #100

Earlier quoted context omitted.

The HTTP GET method is idempotent by specification. Visiting a webpage should not trigger password resets or any other actions by itself. If that's a problem then it's the site's fault for being defective.

So how do you implement a "one click unsubscribe" link in an email? They're on GET requests. You could use JavaScript on the resulting page to then trigger the unsubscribe but bots are now running JavaScript as well.

You show a webpage with an "Unsubscribe" button in it. The button triggers a POST request.

There's also RFC 8058: https://datatracker.ietf.org/doc/html/rfc8058

Re: “Magic links” can end up in Bing search results, rendering them useless

#237

Post by sandermvanvliet »

dTal wrote: post #106
sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

>We ended up adding a custom page to handle e-mail validation so we could handle the situation where the user lands on the page and the address has already been verified. That's a yikes from me! So I can sign up on your service as anyone with an Outlook account, without verification?

No it would still require username/password. This was only verifying the email address was correct.

Re: “Magic links” can end up in Bing search results, rendering them useless

#238

Post by sandermvanvliet »

capableweb wrote: post #175
sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

Wow! I think you just figured out an issue I had while working in a previous company using Auth0, where the authentication token would expire before the user had actually gone there (so the user saw an error page when clicking), but on our side it looked like the user went there but dropped off directly after. Had maybe 1% of the users complaining about this, but we never found the root-cause (we moved to our own aut…

Yw. Took us ages to figure out

Re: “Magic links” can end up in Bing search results, rendering them useless

#239

Post by efreak »

boxed wrote: post #199
9dev wrote: post #173

Earlier quoted context omitted.

You’d assume those have proper robots.txt configuration?

I have a disallow all robots.txt for a production system. Have had from the beginning. Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.

Some services ignore global disallow, but will respect rules explicitly targeted at them.

Re: “Magic links” can end up in Bing search results, rendering them useless

#240

Post by boxed »

efreak wrote: post #239
boxed wrote: post #199

Earlier quoted context omitted.

I have a disallow all robots.txt for a production system. Have had from the beginning. Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.

Some services ignore global disallow, but will respect rules explicitly targeted at them.

I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.
Post reply on HN