I swear I was also thinking of inventing a JSON-based replacement for the IMAP just last week.
JSON Mail Access Protocol Specification (JMAP)
41–50 of 65 posts
Re: JSON Mail Access Protocol Specification (JMAP)
#42Interesting step, but the decisions taken around concurrency seem to assume small-scale, single-user use cases only. I would be the first to admit I haven't given this enough thought yet, but to me it seems that a global lock and MODSEQ make distributed usage etc. very difficult. Isn't this a step backwards?
Re: JSON Mail Access Protocol Specification (JMAP)
#43Lets reinvent everything badly in javascript.
Re: JSON Mail Access Protocol Specification (JMAP)
#44Earlier quoted context omitted.
Specifically, you want a Merkle tree where the final bucket is based on the lowest N bits of the message arrival time, probably at a granularity of a several seconds. In the typical sync case all of the unseen messages will then be clustered in a few buckets and the Merkle tree will be very efficient. For instance, if each bucket contains 8 seconds worth of arrival time, then 2^17 buckets mean they'll wrap around onl…
That's an interesting idea to hash emails to Merkle buckets by arrival time. It may lead to uneven distribution of emails to buckets though, increasing the amount of bandwidth required to sync a bucket in the worst case. There are other optimizations you can bring in when you sync the tree: 1. If you are working your way down the remote's tree and you notice that your local signature is zero for the equivalent remote…
I think the LSB bits of arrival time is actually a really good distribution long-term (remember: we want the distribution to be uneven in the short term, but even in the long term) Maybe you could improve it a little bit by making each bin a prime number of seconds so automated messages sent at the same time every day can hit each bucket eventually. i.e. instead of 8 second granularity, use 7 or 11.
Re: JSON Mail Access Protocol Specification (JMAP)
#45Call me curmudgeonly, why replace a protocol that works...instead roll up a library to provide the functionality... Those things called standards generally find their way through bodies like the IETF. Maybe I missed it while reading on my phone, but I don't see this submitted as an IETF draft...
Read the IMAP specs. If you still don't want to replace it, I can't help you. When I read it the first time, I wanted to hit people in the face.
Re: JSON Mail Access Protocol Specification (JMAP)
#46Earlier quoted context omitted.
I'm in two minds about the UID stuff. There are advantages and disadvantages. But one thing that you MUST have for sensible synchronisation is the global MODSEQ. Otherwise there's no single state you can use to know if there's new data on the server short of (as you do with POP3) fetching the entire UIDL list every time. The way we do this with Cyrus replication is to reinject the message if we get a UID clash - so y…
For anti-entropy in a distributed system, one could use a combination of the following: 1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail. 2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing fr…
(Author of the spec here)
A fixed-size Merkle tree is a great idea if we wanted to sync the entire set of messages between distributed MX destinations. In fact, we should probably consider that approach for our server server replication. However, JMAP is aimed at client server sync. There are a number of different issues to consider here. The client may have limited or no cache (a webmail app for instance must normally start anew each time, and needs to efficiently fetch just what it needs; if we had to wait for a 60GB mailbox to sync every time we loaded our webmail, well, obviously that's not going to work. A mobile mail client often only wants to download the first few items in the mailbox given the current sort order so it can display them to the client. The JMAP spec allows a client to download, say, just the list of the first 50 messages, newest first, in the Inbox (single round trip). Then, at a later point, it can ask for and receive a space-efficient delta update of what has changed within those first 50 messages in the Inbox, regardless of how big the Inbox really is, or what other messages may have been delivered elsewhere (including lower down the Inbox, but outside the top 50). Again in a single round trip. This is really powerful: you can see that by comparing the refresh time of a folder in the FastMail mobile web app with the iOS or Android native mail app (over IMAP); the mobile web app is much quicker.
The other thing to remember is that the metadata about the message is mutable, and needs to be kept in sync as well. Again, the modseq system described in JMAP allows this to happen efficiently. If you have a distributed server system with multiple masters, you will again use a different system to keep those in sync (as they each assign new modseqs, whereas the client does not). However, this too can be dealt with relatively easily: if the modseq for a message differs between two masters, it must be set on both to max(the higher of the two modseqs, the next highest global modseq on the server with the currently lower modseq). This will ensure that both will end up in the same state, and that the client will get refresh its state for that message if it has to, no matter which master it last synced with.
So essentially, there's a difference in what you want in a protocol for server server synchronisation for distributed mail backends compared to client server sync for mail apps. JMAP is focussed on the latter, but tries to make sure it doesn't include assumptions that severely limit the former. The spec does not require a particular format of message ids: they do not have to be based around IMAP's ascending UIDs (although the sync algorithm for partial mailbox listings may be slightly less efficient if not).
Re: JSON Mail Access Protocol Specification (JMAP)
#47Earlier quoted context omitted.
One of the main efficiency gains is in massively reducing the number of round trips required to do a series of operations. This batching of operations is hugely important on high-latency connections, relevant for us in Australia all the time but also of importance on mobile connections the world over. Even 4G networks have relatively high latency, but can do a burst transfer pretty efficiently.
A binary protocol running over persistent socket would have fewer round trips and parsing latency. HTTP and SPDY can't compete with that, because they introduce too much header overhead and complexity.
(Author of the spec here)
The protocol could just as easily go over a WebSocket as HTTP (you'd have to change authentication mechanism, but that would be trivial). The only reason we don't do that at the moment is: 1. Browser support (we support back to IE8, unfortunately) 2. Lack of GZIP compression (particularly of responses from the server); this was being worked on last time I looked, but I haven't checked recently to see if it's been implemented yet. This is crucial for saving bandwidth.
That would get rid of your HTTP overhead; I will put some mention of that in the spec (I think HTTP support should be required, and WebSocket support optional).
As for JSON vs. binary: * There are libraries to use JSON everywhere – binary protocols would require everyone to implement another parser, adding extra difficulty for adoption and a surefire source of bugs. * JSON is readable over the wire, binary is not. Trust me, in the real world, this is a huge advantage for debugging all sorts of things. * JSON can be used trivially in a web app. Apparently this World Wide Web thing might catch on. * The parsing overhead of JSON is not too large; yes it's longer than a binary protocol, but it hasn't been a bottleneck for us, even on mobile.
Re: JSON Mail Access Protocol Specification (JMAP)
#48Interesting step, but the decisions taken around concurrency seem to assume small-scale, single-user use cases only. I would be the first to admit I haven't given this enough thought yet, but to me it seems that a global lock and MODSEQ make distributed usage etc. very difficult. Isn't this a step backwards?
No. See my answer to jorangreef above, but essentially this protocol is for server client sync, but you can still do efficient server server sync for distributed backends.
Re: JSON Mail Access Protocol Specification (JMAP)
#49Interesting. What is the reason for grouping method calls together? Is this to reduce the number of requests sent to the server? If not, is there no way of providing a higher level operation instead of bundled discrete operations? In any case, I think you could have something a lot closer to REST, even if you carry on with grouping calls together. Something like: [ [ "GET", "/messages", { "search": "foo" }], [ "GET",…
(Author of the spec here)
REST is way too slow. The JMAP format massively reduces round trip times, especially when you have sequential operations where you must wait for one to finish before the next can happen. And once you're not doing REST, I don't personally think putting HTTP verbs in the method calls doesn't really make it any clearer, and is just likely to confuse matters.
Re: JSON Mail Access Protocol Specification (JMAP)
#50HTTP itself has an overhead, I'm skeptical changing a persistent IMAP connection into a request/response model with a different serialization format is going to be more efficient, especially given compression (like COMPRESS=DEFLATE) IMAP4rev1 is one thing, but there are many extensions supported in modern imap servers to speed up syncing and mailing, like CATENATE/BURL/CONDSTORE/MULTIAPPEND/MOVE/etc Granted, the IMAP…
One of the main efficiency gains is in massively reducing the number of round trips required to do a series of operations. This batching of operations is hugely important on high-latency connections, relevant for us in Australia all the time but also of importance on mobile connections the world over. Even 4G networks have relatively high latency, but can do a burst transfer pretty efficiently.
> One of the main efficiency gains is in massively reducing the number of round trips required to do a series of operations.
Er…
> 5.5. Multiple Commands in Progress
> The client MAY send another command without waiting for the completion result response of a command, subject to ambiguity rules (see below) and flow control constraints on the underlying data stream. Similarly, a server MAY begin processing another command before processing the current command to completion, subject to ambiguity rules.
IMAP already supports batching.