MBOX-Line: From dave at cridland.net  Thu Jan 20 15:52:09 2011
To: imap-protocol@u.washington.edu
From: Dave Cridland <dave@cridland.net>
Date: Fri Jun  8 12:34:45 2018
Subject: [Imap-protocol] UIDNEXT updates upon new message arrival
In-Reply-To: <4D385AB5.90208@gentoo.org>
References: <4D385AB5.90208@gentoo.org>
Message-ID: <28835.1295567529.285091@puncture>

On Thu Jan 20 15:54:29 2011, Jan Kundr?t wrote:
> when I'm SELECTing a mailbox and have some local state to work with
> (cached UIDs and message parts etc), I'm currently using the  
> UIDNEXT as
> a hint for finding out what happened to the mailbox since the last
> disconnect. My behavior is inspired by a certain document I've read  
> a
> few years ago (and unfortunately I can't find it right now),  it  
> goes
> like this:
> 
> 
You're describing (a slightly broken version of) Polymer's sync  
algorithm.

See https://github.com/dwd/Polymer/blob/master/infotrope/imap.py  
around line 3611, and also line 3800.

> There's a problem with this approach, though, because IMAP servers  
> won't
> send UIDNEXT updates when sending EXISTS that informs about new
> arrivals.


No, but that's okay. If N messages arrive, UIDNEXT must increase by a  
minimum of N. In fact, if N messages arrive, UIDNEXT will in all  
likelyhood increase by precisely N. I can conceive of edge cases  
where it increases by more than N in a reasonable implementation, but  
they're unlikely. So just add one for each new message that you're  
told about during a session. (ie, each time you're told about an  
increase to EXISTS, add the difference to your UIDNEXT).

This is probably easier than your current approach, although that  
looks valid. (Just complicated).

> I do not support QRESYNC yet.

You should do - it's effectively the same sync algorithm, but changed  
to take advantage of the server's knowledge instead - so the  
complicated selection is done by the server.

Polymer doesn't do a binary search, by the way - it fetches chunks of  
1000 UIDs at once, and has an amusing algorithm to decide how many  
chunks to fetch, depending on whether ESEARCH is used (which'll help  
you hugely) and how sparse the UID mapping is. For finding a UID in  
the mapping, it actually does a successive approximation technique,  
on the assumption that the map is uniformly sparse, which works  
pretty well - usually it find the right chunk in its first guess.

For resync purposes, if there have been expunges, it discards and  
refetches just the last block - because generally expunges will  
happen in the recent portion of the mailbox - and if the beginning of  
the newly fetched UID block matches the beginning of the one it's  
replacing, the UID map is now synchronized.

Otherwise, it starts hunting backward through the blocks, several at  
a time, looking for the first unseen expunge, but this is really rare.

Most of this algorithm dates from a time when there was a real  
emphasis on reducing the actual octet count - I don't think that's  
the case so much anymore, so it's probably overkill.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

