Live data from Hacker News

How IMAP works under the hood

blog.lohr.dev

31–40 of 78 posts

Re: How IMAP works under the hood

#31
post #2

Interesting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption. HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.

IMAP had its day in the sun, but the advent of big webmail providers (especially gmail) has killed off the advancement of email clients. Now all major development is focused on trying to recreate Gmail to varying degrees of success. It all ends up internal to one or another corporation so they are just all endlessly reinventing the wheel with IMAP just being relegated to a afterthought front end to some sort of search-based backend.

Actually having a email client software running on your machine is extremely nitch and is mostly in the realm of self-hosters and legacy holdouts that won't let their clients go.

A most advanced modern approach is to just use POP3 to download your emails to a local Maildir and have them indexed there non-destructively. And then sync between your various machines that you want your email accessible using some sort of file sync or P2P solution.

I use notmuch for this. It automatically indexes and tags emails and thus enables much more advanced email management solutions then what can be offered over something like IMAP.

The main advantage of this is that 'folders' are managed virtually. There is no shuffling or copying or editing of emails done normally. I only have to worry about backing up my emails and notmuch config as all the rest can be regenerated relatively quickly.

This is more or less replicating what Gmail and other webmail providers do server side.

Where as the traditional approach shuffling and moving and deleting of emails on some imap server is fairly dangerous and expensive operation. Mistakes can lead to data loss and are often very difficult to reverse.

Re: How IMAP works under the hood

#32
post #27
post #8

I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation…

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard.

For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be successful. Google would do well to adjust their philosophy.

Re: How IMAP works under the hood

#33
post #26

I've been looking to migrate from Exchange-Outlook, but there really aren't any options. There just isn't an open source solution to have an integrated email / tasks / events / contacts, with consistent labels across item types and reminders. With Outlook, i can use a custom view to see every item in a category flagged for follow-up. I can also set a reminder on a contact, or drag an email or contact onto my calendar…

Thunderbird has it all. I don’t like the way thunderbird does it, but I like outlook even less… so thunderbird it is.

Re: How IMAP works under the hood

#34

Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP. IMAP will not work with, say, Google's gmail or Microsoft office365, or AT&T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particul…

> IMAP will not work with, say, Google's gmail or Microsoft office365

Except they do, to some degree. It works well enough that my Thunderbird allows me fetching or moving of mails. Not sure about advanced features like search or server-side filtering, never tried them, but this seems to be a bit more wacky with other clients & servicers too.

> They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it.

True. Gmail at least had a long while application-passwords. I think they changed this only recently? Or are they still a thing?

Re: How IMAP works under the hood

#35
post #11
post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

> I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true. Pretty sure mutt doesn't. It only caches the headers.

> Pretty sure mutt doesn't. It only caches the headers.

Seem like it can cache message bodies:

* http://www.mutt.org/doc/manual/#message-cachedir

* https://neomutt.org/guide/optionalfeatures.html#body-caching

Re: How IMAP works under the hood

#36
post #27

Earlier quoted context omitted.

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be succe…

If an x86 implementation was imperfectly compatible with Intel CPUs, nobody would buy it. Gmail, on the other hand, is a massive market success. It is those who shout that IMAP must be exactly and only whatever mutt+uw was doing in 1997 who are on the wrong side of history.

Re: How IMAP works under the hood

#37
post #27

Earlier quoted context omitted.

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be succe…

It's been an odd running theme for me today that I've misinterpreted posts. Up until your final sentence, I thought that the thesis of your post was:

The standard document may say one thing, but what people do in the real world is the real standard. If your software has issues with the world's most popular IMAP server, you need to adjust your software to be compliant with the standard.

I'm personally more sympathetic to your actual conclusion, but it's odd how often a single argument can be used to support two conflicting beliefs.

Re: How IMAP works under the hood

#38
post #33
post #26

I've been looking to migrate from Exchange-Outlook, but there really aren't any options. There just isn't an open source solution to have an integrated email / tasks / events / contacts, with consistent labels across item types and reminders. With Outlook, i can use a custom view to see every item in a category flagged for follow-up. I can also set a reminder on a contact, or drag an email or contact onto my calendar…

Thunderbird has it all. I don’t like the way thunderbird does it, but I like outlook even less… so thunderbird it is.

[deleted]

Re: How IMAP works under the hood

#39
post #27
post #8

I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation…

> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.

Which concepts are you referring to?

RFC 9051 says that it was "developed for" RFC 822 and that one should now refer to RFC 5322 instead. And RFC 5322 does discuss threading, which is the what the comment you're replying to was talking about, so that concept does appear (by reference) in the IMAP standards.

Are there other parts of the community that implement IMAP using the same "mental model of email" that Gmail is using? Or are you saying that anyone using IMAP with anything but Gmail is an old nerd and a tiny minority?

Re: How IMAP works under the hood

#40
post #33
post #26

I've been looking to migrate from Exchange-Outlook, but there really aren't any options. There just isn't an open source solution to have an integrated email / tasks / events / contacts, with consistent labels across item types and reminders. With Outlook, i can use a custom view to see every item in a category flagged for follow-up. I can also set a reminder on a contact, or drag an email or contact onto my calendar…

Thunderbird has it all. I don’t like the way thunderbird does it, but I like outlook even less… so thunderbird it is.

Thunderbird doesn’t do most of this. It uses tags for emails and categories for events. Also, no way to view all items with same category. Can’t drag contact to calendar or task list. Can’t set specific reminder times. …
Post reply on HN