MBOX-Line: From blong at google.com  Tue Nov 27 14:25:23 2012
To: imap-protocol@u.washington.edu
From: Brandon Long <blong@google.com>
Date: Fri Jun  8 12:34:49 2018
Subject: [Imap-protocol] Re: Suspend/Restore feature proposal
In-Reply-To: <20121127133229.Horde.X3XMkGM8iZ5mPR4tbjUveA1@bigworm.curecanti.org>
References: <20121115215854.Horde.zz6B0O0tmt3ylHiEXzXhQQ4@bigworm.curecanti.org>
	<CABa8R6tHP2My0k2LqT1RzHoLQZA+X_jwUMU0cydm2sAwo8f4Fg@mail.gmail.com>
	<20121116143137.Horde.7Kb3aEW5DhM6CnuA2hAx4Q8@bigworm.curecanti.org>
	<e122fa25-9110-4b36-855d-0e7e273c5805@flaska.net>
	<20121121155417.Horde.ZeW7JqTPNxTAI-hTtrAT-Q9@bigworm.curecanti.org>
	<CABa8R6sJfnA=HYk-fmFfpXki91cne_LeqFyuhDebMGeDdsdj7g@mail.gmail.com>
	<20121121183606.Horde.8DDdRoJ261AbQsAiMPfS4w1@bigworm.curecanti.org>
	<CABa8R6u62m0SkAoLa+K8MgQn8CXKxtv5Hnnt6O1muTZQnFqqoA@mail.gmail.com>
	<20121126152005.Horde.UGsoLB91esWQF9_aGi6UYQ1@bigworm.curecanti.org>
	<CABa8R6tGF8Uj-MEec6vdHrhkYefhiU0QLGtVXafDwOXb3vnFmQ@mail.gmail.com>
	<20121126155721.Horde.a3j2EWFDFOo-xkaflrURBQ2@bigworm.curecanti.org>
	<CABa8R6tOshnrGPGCdb0QToAx8xjWVWY+RBu0Uw8cZ2NH3=TvZw@mail.gmail.com>
	<1354010952.13681.140661158661065.2086C60B@webmail.messagingengine.com>
	<20121127133229.Horde.X3XMkGM8iZ5mPR4tbjUveA1@bigworm.curecanti.org>
Message-ID: <CABa8R6vtOHH24idCZiu+yaudX7i67X4eoE822VwyqgMdaGectg@mail.gmail.com>

On Tue, Nov 27, 2012 at 12:32 PM, Michael M Slusarz
<slusarz@curecanti.org>wrote:

> Quoting Bron Gondwana <brong@fastmail.fm>:
>
>  The interesting part for QRESYNC is remembering "tombstones" for EXPUNGED
>> messages for a while to keep it cheap.
>>
>
> [snip]
>
>
>  If anyone else is interested, I can write up something about how Cyrus
>> implements QRESYNC now, such that it can clean up old records, yet still
>> be efficient for clients most of the time.
>>
>
> I have an interest in this logic, if just for the fact that this issue
> came up recently.
>
> We provide ActiveSync mail syncing through our IMAP library.  (Warning:
> I'm not an ActiveSync expert: this is my knowledge as provided to me by a
> guy who is).  To sync a remote ActiveSync client, it needs to be determined
> which messages have been expunged since the previous sync.  Fantastic if
> QRESYNC is available - we can just use VANISHED so there is no need to keep
> the UID list at the ActiveSync controller layer.
>
> However, we were seeing transactions like the following (on a dovecot
> server):
>
> - HIGHESTMODSEQ as known by activesync client: 53000
> - HIGHESTMODSEQ on IMAP server: 54000
>
> a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53000)
> * VANISHED (EARLIER) 1:37308,37310:40788,40791:**41032,41034:41083
> a OK Fetch completed.
>
> Yikes!  That's over 40,000 UIDs returned.  Sure enough, there seemed to be
> a tipping point where the "expected" VANISHED return - only those UIDs
> actually removed between 53000 and 54000 - was achieved:
>
> a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
> * VANISHED (EARLIER) 1:37308,37310:40788,40791:**41032,41034:41083
> a OK Fetch completed.
> b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
> * VANISHED (EARLIER) 37309,41029:41030,41047:41083
> b OK Fetch completed.
>
> Turns out that dovecot purges old EXPUNGE records every so often from the
> cache.  Discussion was made of ways of possibly improving this behavior,
> and I believe the idea of tombstones/checkpoints came up.
>
> Note: I understand the above VANISHED call is not the most efficient.  I
> told the ActiveSync guy that it would be good practice to also include the
> lowest/highest UID known at a given MODSEQ to make the VANISHED call more
> efficient.  However, this still leaves open the possibility of a large UID
> return range, especially if a user has an old message in their mailbox.
>  e.g. the above example could still possibly look like:
>
> a uid fetch 1000:41100 UID (VANISHED CHANGEDSINCE 53800)
> * VANISHED (EARLIER) 1001:41032,41034:41083
> a OK Fetch completed.
>
> This could be a limitation of ActiveSync, but the problem comes that this
> entire list of UIDs needs to be sent to the remote client.  And if that
> remote client is a mobile device, you are causing a huge amount of traffic
> to be pushed across a wireless connection, and this huge amount of data
> needs to be processed on the remote device.  This is potentially an
> expensive action, both monetary (bandwidth cost), UI-wise (longer time to
> resync), and battery wise if the mobile client is using this.
>
> As a client author I would settle knowing that at most only, say, 1,000
> spurious UIDs will ever be returned from a VANISHED command.  The above
> behavior can be worked around if the list of UIDs is kept at the activesync
> connector level.  But that may not be desirable in any given implementation.
>
> Wondering your theory behind this as a server author.  How much more
> storage does a tombstone regime require on the server?  Is there a way to
> optimize this - i.e. maybe tombstones aren't kept unless/until a client
> actually issues a VANISHED command in a mailbox.  What is a reasonable
> checkpoint/tombstone range?  Is this something that the CHECK command could
> potentially be useful for?


It also depends on the other limits of your backend.  We've been thinking
about a CONDSTORE/QRESYNC system that would be based on our existing
transaction logs, for example, because otherwise trying to shoe-horn the
full IMAP concepts into our existing web optimized backend would be a
pretty expensive.  But since we don't keep all transactions around forever,
we'd have to have a fake floor to our MODSEQ numbers (ie, even if a messge
hasn't changed, it will get a new MODSEQ behind the head).  If your client
didn't sync frequently enough, or was too picky, it could require a
fallback to a full sync (well, full uid/flags sync, not a re-download,
hopefully).

If you design your backend for this stuff from the start, its probably not
too bad, but how many imap servers have been written since QRESYNC was
published, and not just had it shoe-horned into an existing server.

Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20121127/7132b443/attachment.html>
