Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Magic links” can end up in Bing search results, rendering them useless

medium.com

Post reply on HN
Print view View original

61–70 of 243 posts

Re: “Magic links” can end up in Bing search results, rendering them useless

#61

Post by batch12 »

rbut wrote: post #52

I have observed this, but also found that BingBot modifies the query string parameters of your URL. It does this by changing a character of the URL, possibly in an attempt to find new pages? I noticed this because I generate links with a signed token to ensure integrity and started receving invalid token crash reports in Sentry, always from BingBot.. To fix this I had to move the tokens from the query string into the…

I would guess that this is probably done on purpose to avoid tripping one-time-use links. Seems like a good way to hide malware from the scanner though.

Re: “Magic links” can end up in Bing search results, rendering them useless

#62

Post by scandinavian »

rsbadger wrote: post #24
bigtones wrote: post #19

Earlier quoted context omitted.

Can you prove that by linking to a Bing search where one of your pages show up ?

yeh... https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...

The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1...

I don't see that in the robots.txt https://shoprocket.io/robots.txt

User-agent: *

Disallow: /cdn-cgi/l/email-protection

Disallow: /login

Disallow: /register

Disallow: /404

Am I missing something?

Re: “Magic links” can end up in Bing search results, rendering them useless

#63

Post by cube00 »

rsbadger wrote: post #16
andix wrote: post #13

Can’t you block it with robots.txt or some similar method?

It was blocked by robots.txt but Bing chose to ignore it. I even tried "blocking" the URLs in Bing webmaster tools today and this was the response: "Block request denied We found that the URL submitted for block is important for Bing users and hence cannot be blocked through Bing Webmaster Tools. We recommend that the best way to block URLs in this scenario is to add NOINDEX meta-tag to the HTML header of the page."

That is baffling logic. Sure, they think they know best and want to ignore the wishes of the owner of the web site. Why then respect a NOINDEX meta-tag instead of robots.txt?

Re: “Magic links” can end up in Bing search results, rendering them useless

#64

Post by batch12 »

pluc wrote: post #60

I mean you can say that Twitter and Slack for example do it too, any service that generates a preview of your links, they'll crawl the URL you provide whether it's secret (eg sent in a private message) or not. Very very very few will stop at the "og:image" tags and such because why would they discard data about you?

I have observed that twitter's bot hits links within seconds of being tweeted. The traffic comes from several locations, not all twitter ASNs. One interesting source is Apple. Their bot/scanner hits soon after.

Re: “Magic links” can end up in Bing search results, rendering them useless

#65

Post by nopassrecover »

ratg13 wrote: post #58
rsbadger wrote: post #17

Earlier quoted context omitted.

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

Yes, but there is no indication they are publishing it in the search results. The original post is just complaining that the malware scanning is visiting the links. They come to the following conclusion > This effectively makes all one-time use links like login/pass-reset/etc useless. Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thin…

They are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)

Re: “Magic links” can end up in Bing search results, rendering them useless

#66

Post by weberer »

oever wrote: post #43
christophilus wrote: post #36

This would kind of break single-use links, no? That seems like a real nuisance.

The HTTP GET method is idempotent: it should behave the same way on multiple accesses. A single use link, e.g. for resetting a password or confirming a subscription, will usually show a webpage with a form that does a POST. Once that POST has been performed, the single use link is used up. Single use links will mostly have a one-time secret that should not be leaked. Mails that contain such links or any sensitive inf…

How do you send mail to an Outlook user and encrypt it so Microsoft can't snoop on it?

Re: “Magic links” can end up in Bing search results, rendering them useless

#67

Post by Semaphor »

FrenchDevRemote wrote: post #50
Semaphor wrote: post #18

Earlier quoted context omitted.

> AFAIK most email providers (like Gmail) will also visit links. I keep hearing this, but our newsletter system has been using GET unsubscribe links since at least 2007 (but probably longer), and we never found a wave of Gmail users unsubscribing, we still have a lot of them. I wonder if this is simply an urban legend, if Gmail tries to recognize unsubscribe links, or if there is something else going on.

your newsletter system probably ignore bots or some IPs

We do not :) It was never an issue there. Bots get ignored for stats, Google IPs get blocked for Ads (Google seems to think every ad link has to be visited by a ton of bots, our customers actually started complaining about the traffic)

Re: “Magic links” can end up in Bing search results, rendering them useless

#68

Post by baisq »

I remember sending one-time use URLs in emails to customers and they would've expired by the time they clicked them because Outlook was opening them before they did.

Yeah yeah GET is idempotent and I shouldn't do that blah blah. That's not the point.

Re: “Magic links” can end up in Bing search results, rendering them useless

#69

Post by logifail »

Even if Microsoft claim this is about security scanning, isn't it fairly trivial to configure your webserver to serve up different content depending on the User-Agent request header?

BingBot scans the link, gets a dummy page with 'clean' content, Microsoft delivers the email message to the user, user clicks through the link with actual browser, gets phishing / malware content...

Re: “Magic links” can end up in Bing search results, rendering them useless

#70

Post by usrn »

bigtones wrote: post #14

Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.

I just love it when they "scan" password reset links.
Post reply on HN