MBOX-Line: From brong at fastmail.fm  Mon Oct 31 15:24:13 2011
To: imap-protocol@u.washington.edu
From: Bron Gondwana <brong@fastmail.fm>
Date: Fri Jun  8 12:34:46 2018
Subject: [Imap-protocol] BODY.PEEK[section]<origin.size> FETCH response
In-Reply-To: <CABa8R6uckusexAOtq-1-PHd58Sk7Xi0GOwLtwn1HMfiAA52C9w@mail.gmail.com>
References: <8F0DA5FA-FB07-4AFA-9C58-8F0927998343@ghoti.org><alpine.OSX.2.00.1110301823210.9034@hsinghsing.panda.com><CABa8R6uyrHJ6AqQoGfMcLztG-zovaVK4FEFep8ibpYc2-6b7ig@mail.gmail.com><alpine.OSX.2.00.1110311352000.9034@hsinghsing.panda.com><1320096826.9022.140660992904397@webmail.messagingengine.com>
	<CABa8R6uckusexAOtq-1-PHd58Sk7Xi0GOwLtwn1HMfiAA52C9w@mail.gmail.com>
Message-ID: <1320099853.21209.140660992923317@webmail.messagingengine.com>



On Monday, October 31, 2011 3:06 PM, "Brandon Long" <blong@google.com> wrote:
> Ie, we
> implemented COMPRESS when we saw that Thunderbird supported it (and
> used their implementation to fix bugs in ours) and then when Apple saw
> we supported it, they added it to iOS (I'm guessing, but that's what
> it looked like).

Wow - I _can_ change the world!

(that's the only part of Thunderbird I've written)

> For us, it would have been very beneficial if the CAPABILITY command
> had the client telling us which extensions they supported, so we could
> better focus our efforts on the extensions most likely to be useful.
> 
> > I'm actually surprised that no serious attempt has been made to create a
> > competing protocol that's simpler to implement, at both ends. ?I suspect
> > Google could do it themselves with a bit of cooperation from Thunderbird.
> 
> I wouldn't claim to be able to create something better.  I could make
> something simpler with a simpler use case.  The argument may be that
> IMAP tries to do too much, that a two-way syncing protocol would be
> simpler and what most clients these days would use in preference to
> the "on-line, off load work to server" model that say pine wants.  Or,
> that could just be my biases coloring my perception.

Yes, a two way syncing protocol would be very nice.  Also some checksum
stuff to allow you to quickly verify the correctness of the your local
copies.  We expose the sha1 of the messages via IMAP, but of course it's
not standard, so nobody else will use it.

The Cyrus replication protocol contains everything required to regenerate
an exactly matching IMAP image on the remote end, and calculates a
checksum over the entire unexpunged state (we store expunged index records
for a while to support QRESYNC) which can be compared between the two ends.
This allows bandwidth efficient replication - along with the sha1 for
duplicate and copy/rename detection.  In the trivial case we just compare
modseq between master and replica, replicate the records with a higher
modseq plus any messages with UID higher than the remote LAST_UID, and
then compare the SYNC_CRC to make sure the mailbox matches afterwards.

Of course this has been complicated by annotations and conversation ID
(cross folder threading) and there's stuff for supporting sieve, non-owner
seen, subscriptions, etc.

But the basic protocol is fairly "imaplike".

The hardest bit was safely syncing files - I wound up creating an extra
literal syntax that looks like this:

%{partition sha1 size}\r\n
BYTES

The 'partition' is a Cyrus abomination, but is required to efficiently
stream the file to a temporary location on the correct filesystem with
a multi partition installation.

The 'sha1' is mainly used to create the spool filename and to do post-sync
integrity checks.  In theory the sender already re-calculated the sha1
and compared it to the index record before sending, but it doesn't hurt
to re-check.  Our goal is to never replicate corrupted data and break the
other end.

You could avoid all this and have the syntax purely IMAP compatible by
having a more stateful parser of course.  At the moment the replication
engine can parse a full statement off the wire and into memory, knowing
it won't be too giant - and then the files will already be on the right
partition to just hardlink into place.

But I'd be quite interested in having a local client which supported that
protocol and could use it to talk to a remote Cyrus instance.  The mailbox
layer code already asserts all the invarients, so you can never inject
a message with a lower UID than the last one used, or a MODSEQ that isn't
greater than the current HIGHESTMODSEQ on the server.  But allowing you
to explicitly set the target values for them would mean you could keep
two ends exactly in sync with IMAP values.

Whereas at the moment, check what offlineimap does.  It can't even rely
on UIDPLUS, so it has to add a special header to every message it
uploads just so it can tell for sure what UID it got without downloading
the whole bloody thing again.  Or at least it did last time I read the
mutlti-threaded python horror that it was.

Bron ( I have nothing against multi-threading or python... at least
       not until signal handling gets involved )
-- 
  Bron Gondwana
  brong@fastmail.fm


