103 lines
5.0 KiB
Plaintext
103 lines
5.0 KiB
Plaintext
MBOX-Line: From brong at fastmail.fm Mon Oct 31 15:24:13 2011
|
|
To: imap-protocol@u.washington.edu
|
|
From: Bron Gondwana <brong@fastmail.fm>
|
|
Date: Fri Jun 8 12:34:46 2018
|
|
Subject: [Imap-protocol] BODY.PEEK[section]<origin.size> FETCH response
|
|
In-Reply-To: <CABa8R6uckusexAOtq-1-PHd58Sk7Xi0GOwLtwn1HMfiAA52C9w@mail.gmail.com>
|
|
References: <8F0DA5FA-FB07-4AFA-9C58-8F0927998343@ghoti.org><alpine.OSX.2.00.1110301823210.9034@hsinghsing.panda.com><CABa8R6uyrHJ6AqQoGfMcLztG-zovaVK4FEFep8ibpYc2-6b7ig@mail.gmail.com><alpine.OSX.2.00.1110311352000.9034@hsinghsing.panda.com><1320096826.9022.140660992904397@webmail.messagingengine.com>
|
|
<CABa8R6uckusexAOtq-1-PHd58Sk7Xi0GOwLtwn1HMfiAA52C9w@mail.gmail.com>
|
|
Message-ID: <1320099853.21209.140660992923317@webmail.messagingengine.com>
|
|
|
|
|
|
|
|
On Monday, October 31, 2011 3:06 PM, "Brandon Long" <blong@google.com> wrote:
|
|
> Ie, we
|
|
> implemented COMPRESS when we saw that Thunderbird supported it (and
|
|
> used their implementation to fix bugs in ours) and then when Apple saw
|
|
> we supported it, they added it to iOS (I'm guessing, but that's what
|
|
> it looked like).
|
|
|
|
Wow - I _can_ change the world!
|
|
|
|
(that's the only part of Thunderbird I've written)
|
|
|
|
> For us, it would have been very beneficial if the CAPABILITY command
|
|
> had the client telling us which extensions they supported, so we could
|
|
> better focus our efforts on the extensions most likely to be useful.
|
|
>
|
|
> > I'm actually surprised that no serious attempt has been made to create a
|
|
> > competing protocol that's simpler to implement, at both ends. ?I suspect
|
|
> > Google could do it themselves with a bit of cooperation from Thunderbird.
|
|
>
|
|
> I wouldn't claim to be able to create something better. I could make
|
|
> something simpler with a simpler use case. The argument may be that
|
|
> IMAP tries to do too much, that a two-way syncing protocol would be
|
|
> simpler and what most clients these days would use in preference to
|
|
> the "on-line, off load work to server" model that say pine wants. Or,
|
|
> that could just be my biases coloring my perception.
|
|
|
|
Yes, a two way syncing protocol would be very nice. Also some checksum
|
|
stuff to allow you to quickly verify the correctness of the your local
|
|
copies. We expose the sha1 of the messages via IMAP, but of course it's
|
|
not standard, so nobody else will use it.
|
|
|
|
The Cyrus replication protocol contains everything required to regenerate
|
|
an exactly matching IMAP image on the remote end, and calculates a
|
|
checksum over the entire unexpunged state (we store expunged index records
|
|
for a while to support QRESYNC) which can be compared between the two ends.
|
|
This allows bandwidth efficient replication - along with the sha1 for
|
|
duplicate and copy/rename detection. In the trivial case we just compare
|
|
modseq between master and replica, replicate the records with a higher
|
|
modseq plus any messages with UID higher than the remote LAST_UID, and
|
|
then compare the SYNC_CRC to make sure the mailbox matches afterwards.
|
|
|
|
Of course this has been complicated by annotations and conversation ID
|
|
(cross folder threading) and there's stuff for supporting sieve, non-owner
|
|
seen, subscriptions, etc.
|
|
|
|
But the basic protocol is fairly "imaplike".
|
|
|
|
The hardest bit was safely syncing files - I wound up creating an extra
|
|
literal syntax that looks like this:
|
|
|
|
%{partition sha1 size}\r\n
|
|
BYTES
|
|
|
|
The 'partition' is a Cyrus abomination, but is required to efficiently
|
|
stream the file to a temporary location on the correct filesystem with
|
|
a multi partition installation.
|
|
|
|
The 'sha1' is mainly used to create the spool filename and to do post-sync
|
|
integrity checks. In theory the sender already re-calculated the sha1
|
|
and compared it to the index record before sending, but it doesn't hurt
|
|
to re-check. Our goal is to never replicate corrupted data and break the
|
|
other end.
|
|
|
|
You could avoid all this and have the syntax purely IMAP compatible by
|
|
having a more stateful parser of course. At the moment the replication
|
|
engine can parse a full statement off the wire and into memory, knowing
|
|
it won't be too giant - and then the files will already be on the right
|
|
partition to just hardlink into place.
|
|
|
|
But I'd be quite interested in having a local client which supported that
|
|
protocol and could use it to talk to a remote Cyrus instance. The mailbox
|
|
layer code already asserts all the invarients, so you can never inject
|
|
a message with a lower UID than the last one used, or a MODSEQ that isn't
|
|
greater than the current HIGHESTMODSEQ on the server. But allowing you
|
|
to explicitly set the target values for them would mean you could keep
|
|
two ends exactly in sync with IMAP values.
|
|
|
|
Whereas at the moment, check what offlineimap does. It can't even rely
|
|
on UIDPLUS, so it has to add a special header to every message it
|
|
uploads just so it can tell for sure what UID it got without downloading
|
|
the whole bloody thing again. Or at least it did last time I read the
|
|
mutlti-threaded python horror that it was.
|
|
|
|
Bron ( I have nothing against multi-threading or python... at least
|
|
not until signal handling gets involved )
|
|
--
|
|
Bron Gondwana
|
|
brong@fastmail.fm
|
|
|
|
|