MBOX-Line: From jkt at gentoo.org  Thu Jan 20 07:54:29 2011
To: imap-protocol@u.washington.edu
From: =?UTF-8?B?SmFuIEt1bmRyw6F0?= <jkt@gentoo.org>
Date: Fri Jun  8 12:34:45 2018
Subject: [Imap-protocol] UIDNEXT updates upon new message arrival
Message-ID: <4D385AB5.90208@gentoo.org>

Hi,
when I'm SELECTing a mailbox and have some local state to work with
(cached UIDs and message parts etc), I'm currently using the UIDNEXT as
a hint for finding out what happened to the mailbox since the last
disconnect. My behavior is inspired by a certain document I've read a
few years ago (and unfortunately I can't find it right now),  it goes
like this:

if sync_state_not_usable || old.uidValidity != new.uidValidity:
  full_sync()
else:
  if new.uidNext == old.uidNext:
    // certainly no new arrivals
    if new.exists == old.exists:
      // no deletions, either
      sync_just_flags()
    else:
      sync_only_deletions()
  else:
    // Some messages were delivered since the last time,
    // and there's no guarantee that they're still there
    if new.uidNext - old.uidNext == new.exists - old.exists:
      // only new arrivals
      sync_only_additions()
    else:
      sync_generic()

The goal here is to get a list of all UIDs in a mailbox and find out
which messages got deleted (so that I can purge them from the local
cache) and which arrived (so that I can initiate a prefetch for them,
and because I use UID FETCH exclusively, to actually fetch them at all).
I also sync FLAGS to be able to show updated "X new messages" while
messages arrive or are deleted from asynced mailbox.

full_sync() forgets everything from the last time (wiping the cache
completely) and then synces UIDs and FLAGS.

sync_just_flags() fetches all flags.

sync_only_additions() performs "UID SEARCH UID old.uidNext:*" followed
by flags sync.

sync_only_deletions() calls "UID SEARCH ALL" and then synces flags; this
could be optimized to use an algorithm similar to binary search to try
to locate the range where the deletions happened, but that would have
the drawback of increased latency, so I don't do that.

sync_generic() performs in the same way as sync_only_deletions() at this
time (and couldn't be optimized in that manner).

There's a problem with this approach, though, because IMAP servers won't
send UIDNEXT updates when sending EXISTS that informs about new
arrivals. Therefore, when I see the EXISTS which increments the message
count, I walk all the messages I have in my cache at that point,
starting at the message with the highest sequence number and going
backwards to message #1, and stop at the "first" (counting backwards)
message for which I already know the UID. Then I issue a "UID FETCH
highest-known-uid+1:* (FLAGS)" and when processing the untagged FETCH
responses, whenever I store a UID for a message which is higher or equal
than the UIDNEXT I have for the current mailbox, I also set the UIDNEXT
to the UID+1. I'm also listening for ordinary * OK UIDNEXT responses, so
servers which choose to send these updates are handled, too.

I've read various posts to this list about reliability of the UIDNEXT,
but am reasonably sure that this is actually the safe way of using this
value -- the worst thing which could happen is wasting time with
excessively long response to the UID SEARCH command, AFAIK.

I do not support QRESYNC yet.

I'd appreciate if someone could check that the above makes sense and is
actually correct, or advise me about how to change that.

With kind regards,
Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 270 bytes
Desc: OpenPGP digital signature
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20110120/addfc6b1/attachment.sig>
