105 lines
4.9 KiB
Plaintext
105 lines
4.9 KiB
Plaintext
MBOX-Line: From slusarz at curecanti.org Tue Nov 27 12:32:29 2012
|
|
To: imap-protocol@u.washington.edu
|
|
From: Michael M Slusarz <slusarz@curecanti.org>
|
|
Date: Fri Jun 8 12:34:49 2018
|
|
Subject: [Imap-protocol] Re: Suspend/Restore feature proposal
|
|
In-Reply-To: <1354010952.13681.140661158661065.2086C60B@webmail.messagingengine.com>
|
|
References: <20121115215854.Horde.zz6B0O0tmt3ylHiEXzXhQQ4@bigworm.curecanti.org>
|
|
<CABa8R6tHP2My0k2LqT1RzHoLQZA+X_jwUMU0cydm2sAwo8f4Fg@mail.gmail.com>
|
|
<20121116143137.Horde.7Kb3aEW5DhM6CnuA2hAx4Q8@bigworm.curecanti.org>
|
|
<e122fa25-9110-4b36-855d-0e7e273c5805@flaska.net>
|
|
<20121121155417.Horde.ZeW7JqTPNxTAI-hTtrAT-Q9@bigworm.curecanti.org>
|
|
<CABa8R6sJfnA=HYk-fmFfpXki91cne_LeqFyuhDebMGeDdsdj7g@mail.gmail.com>
|
|
<20121121183606.Horde.8DDdRoJ261AbQsAiMPfS4w1@bigworm.curecanti.org>
|
|
<CABa8R6u62m0SkAoLa+K8MgQn8CXKxtv5Hnnt6O1muTZQnFqqoA@mail.gmail.com>
|
|
<20121126152005.Horde.UGsoLB91esWQF9_aGi6UYQ1@bigworm.curecanti.org>
|
|
<CABa8R6tGF8Uj-MEec6vdHrhkYefhiU0QLGtVXafDwOXb3vnFmQ@mail.gmail.com>
|
|
<20121126155721.Horde.a3j2EWFDFOo-xkaflrURBQ2@bigworm.curecanti.org>
|
|
<CABa8R6tOshnrGPGCdb0QToAx8xjWVWY+RBu0Uw8cZ2NH3=TvZw@mail.gmail.com>
|
|
<1354010952.13681.140661158661065.2086C60B@webmail.messagingengine.com>
|
|
Message-ID: <20121127133229.Horde.X3XMkGM8iZ5mPR4tbjUveA1@bigworm.curecanti.org>
|
|
|
|
Quoting Bron Gondwana <brong@fastmail.fm>:
|
|
|
|
> The interesting part for QRESYNC is remembering "tombstones" for EXPUNGED
|
|
> messages for a while to keep it cheap.
|
|
|
|
[snip]
|
|
|
|
> If anyone else is interested, I can write up something about how Cyrus
|
|
> implements QRESYNC now, such that it can clean up old records, yet still
|
|
> be efficient for clients most of the time.
|
|
|
|
I have an interest in this logic, if just for the fact that this issue
|
|
came up recently.
|
|
|
|
We provide ActiveSync mail syncing through our IMAP library.
|
|
(Warning: I'm not an ActiveSync expert: this is my knowledge as
|
|
provided to me by a guy who is). To sync a remote ActiveSync client,
|
|
it needs to be determined which messages have been expunged since the
|
|
previous sync. Fantastic if QRESYNC is available - we can just use
|
|
VANISHED so there is no need to keep the UID list at the ActiveSync
|
|
controller layer.
|
|
|
|
However, we were seeing transactions like the following (on a dovecot server):
|
|
|
|
- HIGHESTMODSEQ as known by activesync client: 53000
|
|
- HIGHESTMODSEQ on IMAP server: 54000
|
|
|
|
a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53000)
|
|
* VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083
|
|
a OK Fetch completed.
|
|
|
|
Yikes! That's over 40,000 UIDs returned. Sure enough, there seemed
|
|
to be a tipping point where the "expected" VANISHED return - only
|
|
those UIDs actually removed between 53000 and 54000 - was achieved:
|
|
|
|
a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
|
|
* VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083
|
|
a OK Fetch completed.
|
|
b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
|
|
* VANISHED (EARLIER) 37309,41029:41030,41047:41083
|
|
b OK Fetch completed.
|
|
|
|
Turns out that dovecot purges old EXPUNGE records every so often from
|
|
the cache. Discussion was made of ways of possibly improving this
|
|
behavior, and I believe the idea of tombstones/checkpoints came up.
|
|
|
|
Note: I understand the above VANISHED call is not the most efficient.
|
|
I told the ActiveSync guy that it would be good practice to also
|
|
include the lowest/highest UID known at a given MODSEQ to make the
|
|
VANISHED call more efficient. However, this still leaves open the
|
|
possibility of a large UID return range, especially if a user has an
|
|
old message in their mailbox. e.g. the above example could still
|
|
possibly look like:
|
|
|
|
a uid fetch 1000:41100 UID (VANISHED CHANGEDSINCE 53800)
|
|
* VANISHED (EARLIER) 1001:41032,41034:41083
|
|
a OK Fetch completed.
|
|
|
|
This could be a limitation of ActiveSync, but the problem comes that
|
|
this entire list of UIDs needs to be sent to the remote client. And
|
|
if that remote client is a mobile device, you are causing a huge
|
|
amount of traffic to be pushed across a wireless connection, and this
|
|
huge amount of data needs to be processed on the remote device. This
|
|
is potentially an expensive action, both monetary (bandwidth cost),
|
|
UI-wise (longer time to resync), and battery wise if the mobile client
|
|
is using this.
|
|
|
|
As a client author I would settle knowing that at most only, say,
|
|
1,000 spurious UIDs will ever be returned from a VANISHED command.
|
|
The above behavior can be worked around if the list of UIDs is kept at
|
|
the activesync connector level. But that may not be desirable in any
|
|
given implementation.
|
|
|
|
Wondering your theory behind this as a server author. How much more
|
|
storage does a tombstone regime require on the server? Is there a way
|
|
to optimize this - i.e. maybe tombstones aren't kept unless/until a
|
|
client actually issues a VANISHED command in a mailbox. What is a
|
|
reasonable checkpoint/tombstone range? Is this something that the
|
|
CHECK command could potentially be useful for?
|
|
|
|
michael
|
|
|
|
|