MBOX-Line: From blong at google.com  Tue Nov 13 16:00:45 2012
To: imap-protocol@u.washington.edu
From: Brandon Long <blong@google.com>
Date: Fri Jun  8 12:34:49 2018
Subject: [Imap-protocol] Re: Efficiently handling sequence numbers
In-Reply-To: <a805dfe9-549d-4230-9655-9668fb1b7319@flaska.net>
References: <CABa8R6v2-NW2L4itj30juL6CEFUC3v9pQG0D0zvqM04UNiB+vA@mail.gmail.com>
	<E121DCDC-75FA-43FB-87A7-EA0EEEEA13B0@iki.fi>
	<CABa8R6sSdHCK9ziPrwc=avaJiHjWtfVtgyBBs=kxSzSkkWJQCQ@mail.gmail.com>
	<8560FE55-3E60-4FCF-9B1A-FB0F66D04295@iki.fi>
	<CABa8R6tdej9ZrNW6EtBXQwuZupsSKP64Fq9fUhJLtJM4oA7NVQ@mail.gmail.com>
	<9B0A7B09-AF1D-41E5-A87E-7BD3A6A05DB2@iki.fi>
	<a805dfe9-549d-4230-9655-9668fb1b7319@flaska.net>
Message-ID: <CABa8R6vALEqiRdf39juUpTJO9zCo73mgwUCTbFhCfv-+fJuRyg@mail.gmail.com>

On Sun, Nov 11, 2012 at 6:49 AM, Jan Kundr?t <jkt@flaska.net> wrote:

> On Friday, 9 November 2012 23:16:19 CEST, Timo Sirainen wrote:
>
>> (Although many IMAP clients fetch 1:* flags after SELECT, but even if
>> IMAP server had no problems with that, the IMAP client would probably be
>> unusably slow so that's not a real problem.)
>>
>
> In Trojita, I always want to establish a fully synchronized view to a
> mailbox, which means either CONDSTORE or QRESYNC or a blind 1:* FLAGS fetch
> upon connecting to a mailbox, yes. At first, this looks like an ugly
> inefficiency, but I believe that it's more or less the only reasonable way.
>

Probably depends on which flags you care about, but the SEARCH method is
certainly less data transfered if you only care about certain flags.  I
estimated that a 100k folder will take around 6MB of data to do the FETCH
1:* (UID FLAGS) response.  Obviously, ESEARCH is even more efficient.


> A graphical MUA typically wants to show a simple statistic like X new, Y
> unread messages. I can get that number through the STATUS command, but what
> shall I do when receving EXPUNGE for a message whose flags are not known --
> shall the number of the unread messages be decreased? Shall I invoke an
> explicit SEARCH NOT SEEN? Shall I just sync the flags in such case? (Note
> that  I cannot use STATUS on an already opened mailbox.)
>

Anyone know what the point of that restriction is?  We've had at least one
client ask us if it was ok, and it works fine on our implementation
(though, its the actual state of the folder, not the possibly out of date
view the client has).

Obviously an ESEARCH (RETURN COUNT) NOT SEEN would work here, but may not
be as optimized (note to self, optimize this case).


> It's true that one can probably come up with a pretty good heuristic like
> "on mailboxes >10k messages, it's unlikely to ever see expunges from the
> beginning, so let's load the flags only on demand" and defer to a full sync
> upon detecting a race. However, because there are extensions (CONDSTORE,
> QRESYNC) which make this synchronization painless and extremely efficient,
> I haven't decided to invest my time in optimizing Trojita for interaction
> with the legacy servers who don't implement these optimizations.
>
> Some of these servers don't even support ESEARCH for an efficient
> compression of the UID ranges which are needed in any client wishing to
> maintain its own offline cache (for essentially the same reason -- you want
> to remove data from the cache upon seeing an EXPUNGE, but in order to do
> that, you have to know the UID of the message which was expunged or come up
> with creative ways of finding out that infromation on demand, which is far
> from trivial -- you could for example ask for the UID of just two messages,
> the one before the message being expunged and the second one right after
> it, but then you have to deal with network disconnects in some complicated
> way, and you would still have to solve this for expunges which have
> happened while offline, between your reconnects).
>
> In short, yep, there are ample opportunities for clients to make the
> interoperability "better", but I somehow don't believe that clients should
> be responsible for this -- there are extensions designed to make this
> problem go away, and the two most widely deployed open source IMAP servers
> have implemented them for quite some time. It's a sad reality that many
> proprietary IMAP server vendors don't bother with ESEARCH, CONDSTORE and
> QRESYNC.
>

ESEARCH - 11/2006
CONDSTORE 6/2006
QRESYNC 3/2008

So, I guess some of these are pretty dated at this point, but client
support is much more recent.  CONDSTORE in particular requires another very
IMAP specific piece of information in the storage layer which made us wait
on more client support before taking the expensive plunge.  Our lack of
ESEARCH support is just an embarrassing result of our hand built parser.

Anyways, these all help with the client synchronization, but don't really
help with the server having to load this information (Well, the client as
well) even if it doesn't sync down to the client.

I'm willing to believe this is only a problem for our ridiculo distributed
implementation, but I really dislike having O(n) operations like this which
are just bound to trip at some number of n.  Maybe I should instead post to
the imap5 list that not having sequence numbers would be a good idea.

Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20121113/8e48c70d/attachment.html>
