JSON Mail Access Protocol Specification (JMAP) • Hacker News Board

Re: JSON Mail Access Protocol Specification (JMAP)

#61

Post by bodyfour » Wed, Jan 29, 2014, 8:35 PM UTC

masklinn wrote: post #51

bodyfour wrote: post #12

Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery…

> Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. I don't understand this criticism, the criticism is usually that IMAP does not require maintaining UIDs (the server can announce UIDNOTSTICKY[0]) > While I'm on the subject, another less-common POP3 extension I liked was "…

> the criticism is usually that IMAP does not require maintaining UIDs

Yes, exactly. UIDs should be sticky, but they should also be allocatable by widely distributed nodes without the need for central coordination.

If your UIDs are 32-bit monotonically-increasing integers then this is impossible. If they are 128-bit random numbers you get it for free. If you prefix them with a timestamp you even get reasonable ordering.

The entire UIDNOTSTICKY problem is a result of IMAP UIDs being so restricted.

I can see why a proposal as complicated as Lemonade failed to get traction, but I find the argument pretty thin. The "attractive nuisance" was the old open-relay SMTP infrastructure. If you were writing a POP client in 1991 you wouldn't think twice about sending mail since you just had to hit a SMTP server (didn't even have to be the "right" one!) and the mail would get delivered. When this all came crashing down far more effort got put into authenticated SMTP (talk about not separating concerns!) than would have been required to just get message submission right in the first place.

Re: JSON Mail Access Protocol Specification (JMAP)

#62

Post by bodyfour » Wed, Jan 29, 2014, 9:05 PM UTC

nmjenkins wrote: post #46

_urga wrote: post #25

Earlier quoted context omitted.

For anti-entropy in a distributed system, one could use a combination of the following: 1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail. 2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing fr…

(Author of the spec here) A fixed-size Merkle tree is a great idea if we wanted to sync the entire set of messages between distributed MX destinations. In fact, we should probably consider that approach for our server server replication. However, JMAP is aimed at client server sync. There are a number of different issues to consider here. The client may have limited or no cache (a webmail app for instance must normal…

> if we had to wait for a 60GB mailbox to sync every time we loaded our webmail, well, obviously that's not going to work

First, the Merkle tree would just be of the IDs not the message contents. I.e. the problem it is trying to solve is for the heavy-weight client that wants to resync its idea of which message UIDs are contained in a mailbox. Which ones it then requests metadata (or full-contents) is up to it.

It's true that for a ephemeral client (like a webapp) you don't even want to get the whole list of 100K messages in your inbox (although you probably will ask the server for a count of them) However, here having a sequential message number hardly helps either. For example, when I click on "sort by date" in my mail client it shows me the messages by sending date, not by what integer IMAP UID they were given.

The problem of "give me the metadata for the top 100 messages sorted by criteria X" is just something that has to be offered by the server. This is no different than features like searching -- if bandwidth were infinite the client could do it, but it's not so you need to do the work near where the data is stored.

The server will have to maintain data structures to make these kinds of queries efficient. The important point however, is that these are conceptually just caches of information that is in the mailstore. Therefore there is no need to keep that synced between far-flung servers.

> The other thing to remember is that the metadata about the message is mutable, and needs to be kept in sync as well

Yes, message data and metadata are separate problems. If two servers both try to update the metadata for message X at the same instant one needs to win (which one is probably not that important) If two servers both try to add new messages to a folder at the same instant they both need to win, ideally without having to coordinate at all in advance.

For the metadata you can use a modseq, a Lamport timestamp, or probably even just a {wallclock time,server ID} tuple. Assuming good time synchronization is reasonable for a mail cluster. If two clients both try to change message flags within 10ms of each other it probably isn't important who wins as long as somebody definitively does.

Re: JSON Mail Access Protocol Specification (JMAP)

#63

Post by brongondwana » Wed, Jan 29, 2014, 9:22 PM UTC

masklinn wrote: post #51

bodyfour wrote: post #12

Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery…

> Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. I don't understand this criticism, the criticism is usually that IMAP does not require maintaining UIDs (the server can announce UIDNOTSTICKY[0]) > While I'm on the subject, another less-common POP3 extension I liked was "…

Mark Crispin was wrong. The underlying model of IMAP is strong, but a lot of the complexity of mail access comes from clients working around things which are either broken or missing in the protocol.

Their reasons aren't excellent, they are some theoretical notion of purity which leads to things like the "MOVE" capability taking years to finally happen, and everyone implementing copy+store+expunge independently and poorly in the meanwhile.

He's dead now, so he can't tell me I'm a feckless Gen X with no clue how to do real protocols - but there's a reason why proprietary protocols like ActiveSync get traction, and it's not because you need to pay a licence fee - it's because they actually solve problems that need solving.

Re: JSON Mail Access Protocol Specification (JMAP)

#64

Post by brongondwana » Wed, Jan 29, 2014, 9:26 PM UTC

kijin wrote: post #21

brongondwana wrote: post #2

This is similar to what we're using at FastMail for our web interface, but simplified and made a bit more generic based on our experiences over the past year. Our plan is to port our web interface across to this spec and implement both a direct backed in Cyrus IMAPd and a standalone implementation which can proxy for existing IMAP servers.

Keep up the good work! 6-year customer here. Notwithstanding some lost features compared to the old UI, your new web interface really lives up to your company's name. Everything is fast and snappy even when I'm accessing it from halfway around the world. But I'm not sure it's a good idea to emphasize the fact that you use JSON. Sure, it's cool right now, but I find it technically more interesting that it replaces IMA…

You need some form of structure unless you want a zillion roundtrips. JSON is cool because it provides the basic structures (scalar, list, map) in a consistent and reliably parsable form. You don't need a context-aware 50 page ABNF to be able to parse it cleanly.

(don't even get me started on trying to write a proxy for IMAP that's aware of the connection state)

Re: JSON Mail Access Protocol Specification (JMAP)

#65

Post by brongondwana » Fri, Jan 31, 2014, 10:49 AM UTC

_urga wrote: post #25

brongondwana wrote: post #14

Earlier quoted context omitted.

I'm in two minds about the UID stuff. There are advantages and disadvantages. But one thing that you MUST have for sensible synchronisation is the global MODSEQ. Otherwise there's no single state you can use to know if there's new data on the server short of (as you do with POP3) fetching the entire UIDL list every time. The way we do this with Cyrus replication is to reinject the message if we get a UID clash - so y…

For anti-entropy in a distributed system, one could use a combination of the following: 1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail. 2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing fr…

I've been waiting a bit to reply to this to see if there's a polite way to answer, but there isn't.

You suffer from a disease, and it's far too bloody common amongst people who design IETF standards.

You're optimising a rare case, where in the worst case you would be re-iding the bulk of the messages delivered in a time period of a few hours, leading clients to have to re-fetch the metadata again (note: it's only the modseq that would change in our design - the UUID is still globally unique and unchaged).

That's no worse than "FETCH 1:* FLAGS" now.

And in exchange for this slight benefit, you would cause EVERY SINGLE CLIENT EVERYWHERE to implement a significantly more complex protocol which requires 4 roundtrips to resync.

Talk about a dog with a very big, very waggy tail.

I'm sorry, but I can't take this suggestion seriously. I've seen too many protocols which are designed with this sort of focus on the worst cases. You have to make sure they don't totally suck - but the important place to optimise is the common case.

...

It turns out there is a way you can gain almost all of the benefits you're looking for - which is delayed MODSEQ allocation. Newly delivered messages can be created with no MODSEQ. When you sync with another either client or server, you allocate a MODSEQ higher than than any which the other end has seen.

Which means you only pay an extra cost if a JMAP peer or client connects at both ends during the link downtime.