Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How IMAP works under the hood

blog.lohr.dev

Post reply on HN
Print view View original

51–60 of 78 posts

Re: How IMAP works under the hood

#51

Post by Aloisius »

jeffbee wrote: post #28
superkuh wrote: post #19

Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP. IMAP will not work with, say, Google's gmail or Microsoft office365, or AT&T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particul…

Struggling to think of a way in which "IMAP will not work with gmail". Please explain.

It can, but it does require doing a lot of Google-specific things (set up a google cloud account, create a consent screen, get a security review, justify your usage of the IMAP API instead of the web APIs to them, find the right scopes, etc) or instruct users to go through multiple screens in their google settings to create an app password.

Google really doesn't want you to use IMAP. They're trying to push everyone to their neutered web apis instead.

Re: How IMAP works under the hood

#52

Post by jeffbee »

Aloisius wrote: post #51
jeffbee wrote: post #28

Earlier quoted context omitted.

Struggling to think of a way in which "IMAP will not work with gmail". Please explain.

It can, but it does require doing a lot of Google-specific things (set up a google cloud account, create a consent screen, get a security review, justify your usage of the IMAP API instead of the web APIs to them, find the right scopes, etc) or instruct users to go through multiple screens in their google settings to create an app password. Google really doesn't want you to use IMAP. They're trying to push everyone t…

You seem to be taking the perspective of an application developer or something like that? Certainly for users all they need to do is roll in with their favorite IMAP client and use it. All of what you said applies not at all to users.

Re: How IMAP works under the hood

#53

Post by [deleted] »

mr_mitm wrote: post #11
camgunz wrote: post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

> I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true. Pretty sure mutt doesn't. It only caches the headers.

[deleted]

Re: How IMAP works under the hood

#54

Post by Aloisius »

philipwhiuk wrote: post #3
therein wrote: post #2

Interesting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption. HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.

The protocol has ossified and been entrenched. In general more efficient usage of IMAP relies on extensions to the protocol. A modern replacement (JMAP) hasn't been adopted by major providers. If you really cared about data transfer size you'd use something like Protobuf.

IMAP 4rev2 merged in a most of the common extensions into the base protocol. Sadly they left out the THREAD, QRESYNC and OBJECTID, though they reference them.

Of course, major providers like Gmail don't support that as well.

Re: How IMAP works under the hood

#55

Post by mjl- »

rmccue wrote: post #8

I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation…

> I started writing a guide to IMAP back when I was working on an email client

I would be very interested in hearing from developers about how they write email clients that need to work with all the servers out there, with the varying levels of IMAP4 (original, rev1, rev2, combinations of at least a dozen extensions) and various levels of buggy behaviour.

I'm assuming a client developer would implement various "profiles". With an "advanced" profile for servers that implement things like CONDSTORE/QRESYNC, and a "very basic" profile that only do the absolute minimum. And probably a profile or two in between. When you encounter unexpected behaviour (eg bad syntax in protocol), you would downgrade the profile for that server for a while (it may get fixed)? If it works like this, I'm curious to the profiles developer choose. If it's not like this, I wonder how client developers work around compatibility issues (can't just keep reconnecting and trying the same if it results in an error!).

Re: How IMAP works under the hood

#56

Post by thesuitonym »

nashashmi wrote: post #17

Can IMAP be used as a file server system? Outlook has this functionality, where files can be stored directly outside of emails.

This is one of those questions where the answer is technically `yes', but for all practical purposes should be considered `no'.

Re: How IMAP works under the hood

#57

Post by chasil »

accrual wrote: post #48

That was a pretty interesting read. I didn't realize one could interact directly with an IMAP server like we can with FTP and telnet + HTTP.

POP is a little simpler, but IMAP is designed for concurrent access by multiple clients to the same account.

What this article doesn't address is OAUTH, which is required more often now.

Re: How IMAP works under the hood

#58

Post by kazinator »

chasil wrote: post #57
accrual wrote: post #48

That was a pretty interesting read. I didn't realize one could interact directly with an IMAP server like we can with FTP and telnet + HTTP.

POP is a little simpler, but IMAP is designed for concurrent access by multiple clients to the same account. What this article doesn't address is OAUTH, which is required more often now.

That's a hugely important use case, especially nowadays. People have mail clients running at the same time from multiple devices; e.g. desktop and mobile. Also multiple browser tabs with an instance of their webmail.

Re: How IMAP works under the hood

#59

Post by Aloisius »

jeffbee wrote: post #52
Aloisius wrote: post #51

Earlier quoted context omitted.

It can, but it does require doing a lot of Google-specific things (set up a google cloud account, create a consent screen, get a security review, justify your usage of the IMAP API instead of the web APIs to them, find the right scopes, etc) or instruct users to go through multiple screens in their google settings to create an app password. Google really doesn't want you to use IMAP. They're trying to push everyone t…

You seem to be taking the perspective of an application developer or something like that? Certainly for users all they need to do is roll in with their favorite IMAP client and use it. All of what you said applies not at all to users.

> roll in with their favorite IMAP client and use it

That's just it. Lots of client developers, especially open source ones, balked.

So to use something like mutt with gmail requires a user go into their google settings, set up 2fa then create an app-specific password. And if a user is on a Google Workspace account with "insecure" passwords turned off, they either have to do all the gcloud/consent/etc. stuff themselves or steal a client secret from another client.

Oauth client secrets aren't really compatible with open source and oauth flows don't work well in terminals. Google's onerous process didn't help and on top of that, using oauth means getting hit by Google's quotas.

Who knows how long Google will support app-specific passwords? Or perhaps they'll start forcing 2fa via their own gmail app every login.

Re: How IMAP works under the hood

#60

Post by camgunz »

mjl- wrote: post #42
camgunz wrote: post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

> You can send lots of requests to a server and you have to tag them so you can match them up with responses later Yes, you can match the final OK/NO/BAD responses with the original command based on the tag. The annoying thing is that each command typically sends most of its data in "untagged responses". And IMAP servers can send unsolicited untagged responses at any time too. It's quite tricky to handle all those re…

Oh shit, of course. The main thing is you can't rely on a double \r\n to terminate a response; you have to track the tag, and yeah it's mega annoying. But multiple connections fixes that, wow.
Post reply on HN