Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How IMAP works under the hood

blog.lohr.dev

Post reply on HN
Print view View original

41–50 of 78 posts

Re: How IMAP works under the hood

#41

Post by Mailtemi »

calvinmorrison wrote: post #22
philipwhiuk wrote: post #3

Earlier quoted context omitted.

The protocol has ossified and been entrenched. In general more efficient usage of IMAP relies on extensions to the protocol. A modern replacement (JMAP) hasn't been adopted by major providers. If you really cared about data transfer size you'd use something like Protobuf.

Well and you could of course server JMAP over something like Protobuf, JSON is used for ease of use but it's not like thats required feature. The bigger benefit of JMAP is it's well designed for dealing with deltas and changes and syncing data requiring less roundtripping than IMAP.

Can confirm that (implemented JMAP). Deltas and asking for the entire mailbox instead of folder by folder is really good. And as a side effect: 1. Less client-side logic for sync. 2. There are many JSON parsers compared to IMAP, making it way easier to use. For example, in C++, you only need a JSON headers-only library, whereas IMAP is meh—only Linux, or clunky usage. Btw, I made one from scratch (IMAP Parser). 3. Goodies like offloading HTTP to the mobile network stack, which supports TLS 1.3 and offline (background) sync, compared to manually extracting certificates from an IMAP connection to validate against the device keychain. It's just 10x easier.

Re: How IMAP works under the hood

#42

Post by mjl- »

camgunz wrote: post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

> You can send lots of requests to a server and you have to tag them so you can match them up with responses later

Yes, you can match the final OK/NO/BAD responses with the original command based on the tag. The annoying thing is that each command typically sends most of its data in "untagged responses". And IMAP servers can send unsolicited untagged responses at any time too. It's quite tricky to handle all those responses correctly in email clients. Indeed, normally you just want to send a command, and get all the data for that command back until the finishing response. IMAP clients typically open multiple connections to isolate commands.

Re: How IMAP works under the hood

#43

Post by mjl- »

jcranmer wrote: post #25
camgunz wrote: post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

> - There's functionality that you basically shouldn't use; the big one is search. Even the specs more or less say "good luck using this". Message sequence numbers. Every folder in IMAP has its emails numbered from 1-N, with no holes, so if you delete a message, everything after it has its message sequence number decremented to close the hole. Except IMAP is multiclient, and the message can be deleted by other connec…

> Any sane client will instead use UIDs for everything

Yes! Since last year there's the experimental UIDONLY extension that allows clients & servers to operate entirely without message sequence numbers. Saves quite a bit of accounting and possibly memory (for large mailboxes). It's a bit surprising the RFC so recent.

> [... thick vs thin clients ...]

For some programmatic email access, IMAP seems quite easy to use. Just write a programmer-readable line of text as command (eg fetch/search/store/etc) and parse the response lines (that's a bit trickier in many cases, probably needing a library, but you may be able to get away without it some of the time) and you're done.

About thick clients: Storage capacity is growing much harder than my email, also on mobile. I would be fine with thick clients that sync all messages. Allows for simpler protocols and simpler implementations. An email access protocol is then not much more than a two-way sync of email message files and associated "dynamic data" like flags/keywords.

But IMAP and JMAP are doing many things at once: synchronization for thick clients, and online access for thin clients. JMAP is specifically targeting the web (thin clients), so what was old is new again (although base IMAP isn't too helpful to (web-based) thin clients either).

Re: How IMAP works under the hood

#44

Post by azhenley »

I've been trying to get approval from Google for the sensitive scopes to use IMAP, and they classified us as needing "CASA Tier 3 Security Assessment". It looks like it is going to be a long, tedious, opaque, and expensive process.

Re: How IMAP works under the hood

#45

Post by mr_mitm »

camgunz wrote: post #18
mr_mitm wrote: post #11

Earlier quoted context omitted.

> I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true. Pretty sure mutt doesn't. It only caches the headers.

Oh, yeah I guess that's what I mean, and then your connection can be used basically (again) like POP

Ah, now I see what you mean. True.

Re: How IMAP works under the hood

#46

Post by donnachangstein »

dbcurtis wrote: post #32
jeffbee wrote: post #27

Earlier quoted context omitted.

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be succe…

If history has taught us anything, it's that Google is happy to willfully ignore, rewrite, and use their market dominance to snuff out any existing standards if they see a way to seize control and make money off something.

Re: How IMAP works under the hood

#47

Post by rmccue »

jeffbee wrote: post #27
rmccue wrote: post #8

I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation…

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

My memory is hazy on it, _but_ from memory, there were some issues around IDs and persistence which don't occur on other servers, as well as the auth being a bit funky. (Gmail also uses its own extension to the protocol for IDs instead of using eg OBJECTID.)

To be clear, I have no opinion on IMAP or what things _should_ do; I certainly was not using IMAP software in the 1990s :) However, trying to implement a client that works across providers does mean trying to operate per the standards, which was a struggle and why I started documenting it :)

Re: How IMAP works under the hood

#49

Post by csb6 »

therein wrote: post #2

Interesting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption. HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.

There is an extension that adds compression. [0] I think most widely-used servers implement it (at least Gmail and Dovecot do)

[0] https://www.rfc-editor.org/rfc/rfc4978.html

Re: How IMAP works under the hood

#50

Post by rmccue »

rprospero wrote: post #37
dbcurtis wrote: post #32

Earlier quoted context omitted.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be succe…

It's been an odd running theme for me today that I've misinterpreted posts. Up until your final sentence, I thought that the thesis of your post was: The standard document may say one thing, but what people do in the real world is the real standard. If your software has issues with the world's most popular IMAP server, you need to adjust your software to be compliant with the standard. I'm personally more sympathetic…

Yeah, agreed! I titled my guide "Practical IMAP" for reason (and almost called it "IMAP As She Is Spoke"). The standards are useful to a point, but actually to jeffbee's point the internet has evolved a lot since then, and how to actually work with modern email is a pretty underdocumented - including by Google themselves on the Gmail-specific parts.
Post reply on HN