Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail is opening and caching URLs within emails without user intervention (2019)

support.google.com

Post reply on HN
Print view View original

151–160 of 271 posts

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#151

Post by raxxorrax »

kubafu wrote: post #138
_wldu wrote: post #3

I built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they provided (as a URL). Token URLs look like this: /validate/email/1d00a5c2648c211befd33f5a8a7cbfab The token is cryptographically strong and disappears after access. It can't be guessed and no one but the email account holder should click it, but I am seeing th…

You can also check for various headers to determine (with quite good accuracy) if a link was clicked by a human or fetched programatically. Here's a list I've accumulated over the years for virtually the same feature as yours: - `sec-fetch-dest` header is present (HUMAN) - `accept` header is present (HUMAN) - `from` header is bingbot(at)microsoft.com (AUTOMATED) - `user-agent` header includes BingPreview (AUTOMATED)…

Also most humans do use browsers, so if you don't have any following requests for resources like scripts, images or just the favicon, you probably got visited by a bot.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#152

Post by raxxorrax »

avereveard wrote: post #65
_wldu wrote: post #3

I built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they provided (as a URL). Token URLs look like this: /validate/email/1d00a5c2648c211befd33f5a8a7cbfab The token is cryptographically strong and disappears after access. It can't be guessed and no one but the email account holder should click it, but I am seeing th…

I've had a lot of grief from a few users' Exchange doing it (likely as part of some anti phishing plugin of sorts), to the point we changed validation links from one time to sort lived.

Don't know why you are downvoted. Many corporations and institutions employ sandboxes to check mails and the links contained in them. This is a standard security practice by now.

So a link that is only valid once would be affected. Restricting the validity by time is a good way to solve this while still maintaining decent security.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#153

Post by obuda »

_wldu wrote: post #3

I built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they provided (as a URL). Token URLs look like this: /validate/email/1d00a5c2648c211befd33f5a8a7cbfab The token is cryptographically strong and disappears after access. It can't be guessed and no one but the email account holder should click it, but I am seeing th…

Another idea is to have 3 links, where only one is visible:

  https://example/com/token?forBots
  https://example.com/token
  https://example.com/token?forBots
Hopefully any automated systems will open the first or last link first, so that you can save the request info and filter based on that. In case requests come out of order, you can always add a small delay to the "human" link before responding.

I haven't yet gotten to implementing any of the authentication on my current project, so I might be missing something really basic.

The next best thing is to set a cookie when requesting the magic link, but the downside (or upside?) is that it will be valid only for the browser it was requested with.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#154

Post by red_admiral »

Wasn't there an issue like this years ago in something called Google Web Accelerator (I think it was a toolbar for IE).

Theory: it preloads links in the background.

Practice: some old bb showed a (delete) link after each post and a (ban) link, among others, next to each user if you were logged in as administrator. All of these sent GET requests because some developer hadn't read that part of the standards, and there was no "are you sure?" prompt either.

What I think is happening here is that gmail is scanning the content of each link in an e-mail for some subset of {malware, fraud, phishing, child abuse, other bad stuff}. This is a feature if you're a non-techy user who clicks on phishing links, I suppose?

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#155

Post by prepend »

sleavey wrote: post #78
sxp wrote: post #25

Earlier quoted context omitted.

And make sure the action is a POST instead of a GET. GETs should never modify important state.

This is the correct answer. Just because the norm is to embed verification hashes in URLs to be clicked, doesn't mean it's the right way for it to be done. Why not send a short random code by email for the user to then copy into the sign-up form they were in the process of filling in?

> user to then copy into the sign-up form

Extra steps are hard and boring and people don’t want to do them.

I consider myself a savvy user and I want to click a link. Not click a link, then look up a code from the email, then paste, then click submit.

I’d live with having to manually click “I’m sure I want to unsubscribe” or something.

This is most annoying when the site wants me to type in my email address to unsubscribe. I have lots and lots of different email addresses that funnel into a single one. When the site doesn’t put my address in the “To” field, I dont know who they sent to.

Services should be respectful of users time.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#156

Post by pwdisswordfish0 »

geofft wrote: post #97
pwdisswordfish0 wrote: post #69

Earlier quoted context omitted.

> to open one secret link "Secret link" is an oxymoronical concept. Resource identifiers are exactly that: identifiers. They're not private names, and any design that relies on keeping them secret is inherently flawed. If it's accessible on the openly resolvable web, then the content needs to be treated as if it's public. If your use calls for authentication or authorization, then actually use an authentication or au…

They are in fact private names, because they're unknown to the public. This is in fact an authentication system. Yes, the public could guess a 128-bit random value and log in - but that's no different from the ability of the public to guess your password, or your session cookie, or your SSL session state, or whatever. Every authentication mechanism is based on "There is a high-entropy value, and nobody but the author…

You're doing some subtle jiu jitsu and extracting a lot of benefit from responding to the previous message as if it said "if the names are not known to the public, then[...]". It does not.

Resource identifiers, on the web[1], are not private names—not even by virtue of the fact they were communicated over a private channel—and they need to be treated as public, full stop. URLs are not private names, simply because of what they are.

> It makes no difference from a theoretical standpoint [...] whether the high-entropy value is sent to the server as part of the URL or via a header or via POST data.

It makes no difference from an information theoretic standpoint. There is no reason, however, to narrowly consider the information content and its entropy and declare that you are done. From an information architecture standpoint, there is a difference.

> But that's no different from, like, the need to not expose your cookies

It is different, for the reasons above.

(Every entropy-based cryptographic protocol also begins with observations how hard it is to do something in practice, and is then founded on exploiting those side effects. To describe a system and then wave away concerns that it is merely unfit "from a practical standpoint" makes it a failure of a design. It is fundamentally at odds with not just the evaluation criteria that protocols fit for use are measured against, but from which they are born.)

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#157

Post by prepend »

duckmysick wrote: post #118
ace2358 wrote: post #95

Earlier quoted context omitted.

Kinda. I often read my email on my phone while working on my desktop. (Or visa versa). In these situations, a code is always better. I hate the links personally.

How many times having to click a link (instead of entering a code) stopped you from finishing a sign-up process?

It’s pretty easy to measure. I had a site with a verification step. And we would see like 20% drop off of people who clicked on the link but never confirmed. Not sure why. We didn’t have them copy and paste anything, just click a confirm button.

Switching to no confirm obviously changed this to 0% drop off of people who clicked the link, but the number of people who clicked was the same.

It was curious to me why people wouldn’t go through with the confirmation step, but never learned why. We just learned that for some reason more people click once instead of twice.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#158

Post by toshk »

miken123 wrote: post #124
harg wrote: post #96

Earlier quoted context omitted.

You’d need to send the user to the verify page and populate a form with their token from the url. Then submit the form, either automatically or by getting the user to manually hit a button.

You should not submit the form automatically. Tools like Microsoft O365 ATP run any links in an emulated browser with Javascript support. These will, in some cases, happily autosubmit the form for you.

We've done this for over 2 years now, and over 200k users never ran into this. Also not within government users.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#159

Post by duckmysick »

tonypace wrote: post #123
duckmysick wrote: post #118

Earlier quoted context omitted.

How many times having to click a link (instead of entering a code) stopped you from finishing a sign-up process?

Nobody remembers the exact moment they stopped thinking about something because it was easier not to. Ragequitting is one way to exit a process, but just not going to the next step from distraction is surely more common.

I'm asking because often when I talk to people about things they hate, they end up admitting it's not that big of a deal. The annoyance is minor enough they don't look for alternatives or abandon whatever they were doing.

The original discussion was about clicking links vs reading and entering the code in sign-up confirmations. The former takes less steps and is easier to complete. Power users with unusual habits might disagree. But if they complete the sign-up anyway, it makes more sense to focus on regular users.

Re: Gmail is opening and caching URLs within emails without user intervention (2019)

#160

Post by mat0 »

shadowgovt wrote: post #148
motoboi wrote: post #129

Earlier quoted context omitted.

Well, this is why your email provider should not open your links for you. Use a different email provider instead.

I think I'm going to continue using the one that is pre-opening links that should be idempotent so that it can check them against its heuristics for spam or phishing. That's been really nice to have. And I'll instead refrain from using sites that inappropriately provide bare get URLs that are really state-mutating booby traps in disguise.

Lol, the nerve of the other reply: "stop using the most widely used email provider in the world". I liked your reply better
Post reply on HN