Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How IMAP works under the hood

blog.lohr.dev

Post reply on HN
Print view View original

61–70 of 78 posts

Re: How IMAP works under the hood

#61

Post by XorNot »

jeffbee wrote: post #36
dbcurtis wrote: post #32

Earlier quoted context omitted.

That is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be succe…

If an x86 implementation was imperfectly compatible with Intel CPUs, nobody would buy it. Gmail, on the other hand, is a massive market success. It is those who shout that IMAP must be exactly and only whatever mutt+uw was doing in 1997 who are on the wrong side of history.

It's a free email account, it is not at all clear that "weird IMAP" is core to that success?

Certainly I moved away to Fastmail, which has better IMAP support (but mostly because Google having full control of my email address was becoming too big of a risk and Google Apps is expensive for your own domain).

Re: How IMAP works under the hood

#62

Post by csb6 »

mjl- wrote: post #55
rmccue wrote: post #8

I started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation…

> I started writing a guide to IMAP back when I was working on an email client I would be very interested in hearing from developers about how they write email clients that need to work with all the servers out there, with the varying levels of IMAP4 (original, rev1, rev2, combinations of at least a dozen extensions) and various levels of buggy behaviour. I'm assuming a client developer would implement various "profi…

As someone currently working on an IMAP client, my strategy has been to start by implementing the base IMAPrev2 specification and checking that it still works with a rev1 server. RFC 9051 has guidance on how to handle servers that use older IMAP protocol versions and is largely backwards compatible.

The goal is to have the client work without any extensions. E.g., if the server doesn’t have the UNSELECT capability it falls back to selecting a nonexistent mailbox (which has the same effect). Obviously there are some very useful extensions that the client needs to support, but those can be viewed as optimizations to reduce network traffic or add extra features if the server supports them.

Re: How IMAP works under the hood

#63

Post by userbinator »

camgunz wrote: post #10

I've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e.…

You can group all this under the a heading of "we thought people would use this over telnet"

No, the grammar is too convoluted for that. It's more like "we thought text-based protocols are better".

POP3 and SMTP are relatively easy to use manually. IMAP is not.

Re: How IMAP works under the hood

#64

Post by XorNot »

slightwinder wrote: post #34
superkuh wrote: post #19

Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP. IMAP will not work with, say, Google's gmail or Microsoft office365, or AT&T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particul…

> IMAP will not work with, say, Google's gmail or Microsoft office365 Except they do, to some degree. It works well enough that my Thunderbird allows me fetching or moving of mails. Not sure about advanced features like search or server-side filtering, never tried them, but this seems to be a bit more wacky with other clients & servicers too. > They have each implemented their own proprietery out-of-band authenticati…

OAuth works fine for major providers in Thunderbird...unless you're using hard tokens with pin number requirements right now (the entry form doesn't display).

Re: How IMAP works under the hood

#67

Post by azhenley »

isaachinman wrote: post #66
azhenley wrote: post #44

I've been trying to get approval from Google for the sensitive scopes to use IMAP, and they classified us as needing "CASA Tier 3 Security Assessment". It looks like it is going to be a long, tedious, opaque, and expensive process.

What are you building?

An email client for Gmail.

Re: How IMAP works under the hood

#68

Post by calvinmorrison »

Mailtemi wrote: post #41
calvinmorrison wrote: post #22

Earlier quoted context omitted.

Well and you could of course server JMAP over something like Protobuf, JSON is used for ease of use but it's not like thats required feature. The bigger benefit of JMAP is it's well designed for dealing with deltas and changes and syncing data requiring less roundtripping than IMAP.

Can confirm that (implemented JMAP). Deltas and asking for the entire mailbox instead of folder by folder is really good. And as a side effect: 1. Less client-side logic for sync. 2. There are many JSON parsers compared to IMAP, making it way easier to use. For example, in C++, you only need a JSON headers-only library, whereas IMAP is meh—only Linux, or clunky usage. Btw, I made one from scratch (IMAP Parser). 3. Go…

It's designed well, from the ground up with a lot of lessons learned from some pretty serious email junkies. Anyone hemming and hawing about JSON being selected as the transport encoding or whatever is just raising silly points. You could do JMAP over protobuf too probably if you cared, but the whole point (IMO) is to get the hell away from unstructured protocols.

Re: How IMAP works under the hood

#69

Post by Mailtemi »

calvinmorrison wrote: post #68
Mailtemi wrote: post #41

Earlier quoted context omitted.

Can confirm that (implemented JMAP). Deltas and asking for the entire mailbox instead of folder by folder is really good. And as a side effect: 1. Less client-side logic for sync. 2. There are many JSON parsers compared to IMAP, making it way easier to use. For example, in C++, you only need a JSON headers-only library, whereas IMAP is meh—only Linux, or clunky usage. Btw, I made one from scratch (IMAP Parser). 3. Go…

It's designed well, from the ground up with a lot of lessons learned from some pretty serious email junkies. Anyone hemming and hawing about JSON being selected as the transport encoding or whatever is just raising silly points. You could do JMAP over protobuf too probably if you cared, but the whole point (IMO) is to get the hell away from unstructured protocols.

Yep, there's always someone ranting about JSON inefficiencies, which I don't observe at all :)

Re: How IMAP works under the hood

#70

Post by Avamander »

superkuh wrote: post #19

Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP. IMAP will not work with, say, Google's gmail or Microsoft office365, or AT&T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particul…

> Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP.

Not really true. It's usually the client implementations that violate the standard in some way or another, like Outlook. But there are way more bespoke rare clients that have poor implementations.

> They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it.

Well, no. They have implemented OAuth and that's not proprietary. They do it because plain login has massive downsides.

Post reply on HN