31–40 of 65 posts
Call me curmudgeonly, why replace a protocol that works...instead roll up a library to provide the functionality... Those things called standards generally find their way through bodies like the IETF. Maybe I missed it while reading on my phone, but I don't see this submitted as an IETF draft...
Any new access protocol for mail should be designed to work over persistent secure socket, not over HTTP. Additionally, there should be a minimum of work imposed on the server by the client.
Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery…
In any case, I think you could have something a lot closer to REST, even if you carry on with grouping calls together.
Something like:
[
[ "GET", "/messages", { "search": "foo" }],
[ "GET", "/mailboxes", { "etag": "bar" }]
]
Earlier quoted context omitted.
I'm in two minds about the UID stuff. There are advantages and disadvantages. But one thing that you MUST have for sensible synchronisation is the global MODSEQ. Otherwise there's no single state you can use to know if there's new data on the server short of (as you do with POP3) fetching the entire UIDL list every time. The way we do this with Cyrus replication is to reinject the message if we get a UID clash - so y…
For anti-entropy in a distributed system, one could use a combination of the following: 1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail. 2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing fr…
For instance, if each bucket contains 8 seconds worth of arrival time, then 2^17 buckets mean they'll wrap around only every ~12 days. So if you are syncing after a day away there will be only 8% of the buckets potentially dirty (even if you get an incredible flood of mail!)
Having the bucket use based on time also allows you to avoid some of the roundtrip costs when doing frequent syncing. In a single request, the client can speculatively send its hashes of buckets that are likely to have changed since its last sync time along with its root hash. The server finds which buckets changed, sending those new bucket contents -- it can then see that the top level hash must now agree on both sides and you're done in one RT.
Of course it's possible that other buckets have changed in rare edge cases; in this case the top hashes will not match and you have to do the full Merkle descent to resync.
Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery…
I'm in two minds about the UID stuff. There are advantages and disadvantages. But one thing that you MUST have for sensible synchronisation is the global MODSEQ. Otherwise there's no single state you can use to know if there's new data on the server short of (as you do with POP3) fetching the entire UIDL list every time. The way we do this with Cyrus replication is to reinject the message if we get a UID clash - so y…
Also, I strongly believe that per-message UUIDs should always start with the delivery time. That way as long as your servers are all reasonably synchronized you can get reasonable "sort by folder order" behavior without any sequential ID.
You're right that the modseq issue is hard, but the Merkle tree suggestion from @jorangreef can address that. Refer to my reply to him for more details.
Unfortunately it includes my biggest gripe about IMAP -- the requirement that messages are given a server-side message number. Maybe breaking with IMAP on this would make interop too hard though. This requirement is convenient for the client but makes implementing robust servers harder than it need be. If instead each message was identified by a {timestamp,UUID} tuple then multiple MX servers could do final delivery…
Some IMAP servers allow you to use a send folder to inject mail to be sent. I think that's a pretty reasonable way to go about it with the way IMAP works, but afaik no client implements it. Agreed on the messageid thing. I frankly think that IMAP is too messy to try too hard for interop with. Using a relatively similar data model and being able to work correctly with a maildir should be considered enough. But I love…
A proper mail submission command would preserve parity with SMTP by keeping envelope and headers distinct. I just want to leverage the fact that I've already auth'ed the user and shouldn't have to redo that to a separate server.
Also, presumably one of the advantages to the REST+JSON design of JMAP is to make it easy for javascript clients to use it. Since they can't do direct SMTP anyway there will presumably be some REST service that they will use for submission.
It just seems that it should be part of the same standard. Most mail readers are also mail senders.
Earlier quoted context omitted.
For anti-entropy in a distributed system, one could use a combination of the following: 1. Get everything created since last sync. A created index (as opposed to updated index) would not need to be resorted for subsequent updates. This would get only new mail. 2. A fixed-size Merkle tree, 8 blocks per level, 7 levels deep, with around 262144 blocks at the base. This would be the best tree configuration for syncing fr…
Specifically, you want a Merkle tree where the final bucket is based on the lowest N bits of the message arrival time, probably at a granularity of a several seconds. In the typical sync case all of the unseen messages will then be clustered in a few buckets and the Merkle tree will be very efficient. For instance, if each bucket contains 8 seconds worth of arrival time, then 2^17 buckets mean they'll wrap around onl…
There are other optimizations you can bring in when you sync the tree:
1. If you are working your way down the remote's tree and you notice that your local signature is zero for the equivalent remote's node signature, then you know that your entire subtree is empty, and you can short-circuit and start downloading entire buckets.
2. Conversely, if you are working your way down the remote's tree and you notice that your local signature is present but the equivalent remote's node signature is zero, then you know that your entire subtree has data, but the remote's entire subtree is empty, and you can short-circuit and start uploading entire buckets.
3. As you work your way down sections of the tree, you can start to build an idea of the average email to bucket ratio. If several buckets are only likely to contain at most one email, then you can short-circuit again.
Earlier quoted context omitted.
A binary protocol running over persistent socket would have fewer round trips and parsing latency. HTTP and SPDY can't compete with that, because they introduce too much header overhead and complexity.
Persistent connections are tough on patchy mobile networks. Parsing latency isn't really an issue; all this stuff is IO bound anyway. CPU cycles are cheap.
A sure-fire way to decrease latency is to send as little as possible.
An empty default browser HTTP request is likely to cost you around 500 bytes in headers, before you have added any headers or data. Contrast that with a binary command which can be 1 byte. And then multiply this by hundreds of requests. CPU cycles on mobile are not cheap. It makes no sense to parse 500 bytes (and then GC this later) when you don't have to.
Earlier quoted context omitted.
Persistent connections are tough on patchy mobile networks. Parsing latency isn't really an issue; all this stuff is IO bound anyway. CPU cycles are cheap.
If persistent connections are patchy, then HTTP is no further ahead. A sure-fire way to decrease latency is to send as little as possible. An empty default browser HTTP request is likely to cost you around 500 bytes in headers, before you have added any headers or data. Contrast that with a binary command which can be 1 byte. And then multiply this by hundreds of requests. CPU cycles on mobile are not cheap. It makes…