Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

Post reply on HN
Print view View original

151–160 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#151

Post by elric »

jeroenhd wrote: post #144

Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…

That's a bit of a narrow view on this problem. When sending a link to someone, you expect that someone to view the link. Not some random mail service. Who gave the mail server permission to access the page? What if it contains copyrighted material? What if it's one of the millions of pages which don't follow the HTTP design philosophy to the letter?

This is a can of worms.

Re: “Magic links” can end up in Bing search results, rendering them useless

#152

Post by duxup »

sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

I had a customer who had some sort of software that followed the link in the email we sent (no big deal so far), and THEN would follow every link and button on that page.

We had a handy quick decline and accept button on there so they were auto declining things…

I didn’t hate email until I got into web development….

Re: “Magic links” can end up in Bing search results, rendering them useless

#153

Post by progman32 »

Our company occasionally does "test phishes" to see how well people resist them. Every time, some of our most security minded engineers end up on the "clicked on the malicious link" lists, when all they did was forward the message to IT to report the phish. I'm wondering if the bingbot leak is the reason.

Re: “Magic links” can end up in Bing search results, rendering them useless

#154

Post by chexum »

sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

IMHO it is clearly a wrong assumption on the side of any such sender. A verification link should have clear definite actions for the user receiving it:

- It's me, let me confirm my address

- I never signed up for this heap of diamonds

Whenever I (not even a bot) click or follow a link from my mailbox, by accident or on purpose, I don't expect that to validate an account for anyone else, but me, intentionally, using a password I know.

Re: “Magic links” can end up in Bing search results, rendering them useless

#155

Post by names_are_hard »

elric wrote: post #151
jeroenhd wrote: post #144

Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…

That's a bit of a narrow view on this problem. When sending a link to someone, you expect that someone to view the link. Not some random mail service. Who gave the mail server permission to access the page? What if it contains copyrighted material? What if it's one of the millions of pages which don't follow the HTTP design philosophy to the letter? This is a can of worms.

> Who gave the mail server permission to access the page?

The recipient of the email, or their employer's IT department that is paying another company for mail services.

If you send me an email with a link then I do believe I have the right to send that link to a third party service that can validate that it's not malicious. If I decide to sign up for a mail service that promises to protect me from phishing emails, then I [0] expect said service to read the emails I receive and examine the links within them. I would be upset if the service used the info I share with them for purposes other than keeping me safe, though.

I readily admit that I have, at various points in my life, signed up for services without reading the entire TOS that I agreed to. I try to choose companies that I feel I can trust to not abuse me too much, and sometimes I avoid certain services because I don't trust the company behind them enough to respect my privacy.

[0] I acknowledge that not everyone is as knowledgeable as me, and many people might not realize that this is how the protection works. So if the argument is more education, I'm in favor.

Re: “Magic links” can end up in Bing search results, rendering them useless

#156

Post by jeroenhd »

rsbadger wrote: post #145
jeroenhd wrote: post #144

Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. I don't see the problem here, all services need to do is add a page that's says "welcome back, $Username, click here to log in!" that sends a POST request to do any serious confirmation without breaking any specifications. Microsoft claims…

I’m fairly certain they are. My links ended up indexed in Bing search results. The only place they were ever rendered was in private emails to users. Bing should not be indexing that.

You're right, it shouldn't. It's possible that they're fetching these URLs from their customers' browsing history and submitting those (external submissions follow different crawling rules, sometimes bypassing robots.txt). Bing's webmaster information says so, at least: https://www.bing.com/webmasters/help/webmasters-guidelines-3...

For a bit of added "fun", Google will do the same, but if you add a page to robots.txt and set noindex then they won't process the noindex parameter and external indexing sources might still generates search results: https://developers.google.com/search/docs/advanced/crawling/...

Re: “Magic links” can end up in Bing search results, rendering them useless

#157

Post by jkaptur »

duxup wrote: post #152
sandermvanvliet wrote: post #97

I've had to deal with this with e-mail verification links and Auth0. The user clicked the link after getting it in their mailbox but then Auth0 throws up an error page because the e-mail address has already been verified (by Outlook scanning). The problem becomes worse if for some reason the mail ends up in the junk mail folder so the user thinks they've never received the mail but when you check it looks like the e-…

I had a customer who had some sort of software that followed the link in the email we sent (no big deal so far), and THEN would follow every link and button on that page. We had a handy quick decline and accept button on there so they were auto declining things… I didn’t hate email until I got into web development….

The crawler followed buttons on forms? And sent POST requests? Yikes..

Re: “Magic links” can end up in Bing search results, rendering them useless

#158

Post by martin_drapeau »

In the B2B SaaS where I work we started using single use codes to log in for certain account types (non-admins). No password. We send you an email or an SMS with a 6-digit number. Copy/paste it to log in. Very much like 2FA except there is no password. The session lasts 30 days. The user can disconnect of course.

Curious what HN readers think. Is this secure? Sufficient?

Re: “Magic links” can end up in Bing search results, rendering them useless

#159

Post by duxup »

jkaptur wrote: post #157
duxup wrote: post #152

Earlier quoted context omitted.

I had a customer who had some sort of software that followed the link in the email we sent (no big deal so far), and THEN would follow every link and button on that page. We had a handy quick decline and accept button on there so they were auto declining things… I didn’t hate email until I got into web development….

The crawler followed buttons on forms? And sent POST requests? Yikes..

Yup. It was a pain. I have no idea who wrote that and thought it was a good idea…

It was a super basic web form too. Probably the most html markup standard thing we have. Nothing strange about it that could have triggered some sort of strange behavior.

Post reply on HN