MBOX-Line: From mrc at CAC.Washington.EDU  Tue Jul 31 19:23:48 2007
To: imap-protocol@u.washington.edu
From: Mark Crispin <mrc@CAC.Washington.EDU>
Date: Fri Jun  8 12:34:40 2018
Subject: [Imap-protocol] Re: [Alpine-alpha] problem with "expunge"
In-Reply-To: <Pine.WNT.4.64.0707182348130.2684@name-hqai7p9nl5>
References: <alpine.WNT.0.999.0707181554170.3328@PM-100140.ads.qub.ac.uk>
	<alpine.WNT.0.999.0707180959280.2576@Shimo-Tomobiki.Panda.COM>
	<Pine.WNT.4.64.0707182348130.2684@name-hqai7p9nl5>
Message-ID: <alpine.LRH.0.999.0707311903340.10533@shiva1.cac.washington.edu>

Thank you for reporting this problem and sending me a debug log.  I 
apologize for the delay in replying, due to being on vacation.

The underlying problem is with the Exchange server, which is not 
implementing the IMAP protocol correctly.  The code in Alpine, and the 
c-client library called by Alpine, was insufficiently defensive against 
invalid protocol from the server.

I am bcc'ing this message to a colleague at Microsoft who used to work on 
the IMAP code in Exchange and knows who needs to act on this problem.  I 
am also cc'ing the IMAP protocol mailing list, so that other IMAP software 
developers are aware of the issue.

From your log, early in the session, the Exchange server sent the 
following:

IMAP DEBUG 23:44:14.281: * 609 EXISTS

which indicates that there are 609 messages in the mailbox.  Later, when 
you go to expunge the following happens:


IMAP DEBUG 23:44:35.265: 00000009 EXPUNGE
IMAP DEBUG 23:44:35.343: * 608 EXISTS
IMAP DEBUG 23:44:35.343: * 609 EXPUNGE
IMAP DEBUG 23:44:35.343: 00000009 OK EXPUNGE completed.

This is a big whoops.  The
 	* 608 EXISTS
from the server is invalid, as it shrinks the mailbox.  Only an EXPUNGE 
can do this.  EXISTS can only be sent with a size that is greater than or 
equal to the current size.  Unfortunately, the c-client library innocently 
accepted the shrunken value.

Then, along comes the
 	* 609 EXPUNGE
which refers to a message that does not exist (since the EXISTS said that 
there are only 608 messages).  That's why you got the "Unknown message 
data: 609 EXPUNGE"; that was defensive code added earlier to defend 
against expunging non-existent messages.

A correct scenario would be:
 	00000009 EXPUNGE
 	* 609 EXPUNGE
 	* 608 EXISTS
 	00000009 OK EXPUNGE completed.
Here, the EXPUNGE removes message 609, and the EXISTS simply confirms that 
there are only 608 messages.  An equally valid scenario would be:
 	00000009 EXPUNGE
 	* 609 EXPUNGE
 	00000009 OK EXPUNGE completed.

Although the code in c-client protected against the bogus EXPUNGE, it 
didn't protect against the bogus EXISTS.  Other parts of Alpine still used 
the 609 value (remember, the EXPUNGE got disregarded due to the EXISTS). 
Thus Alpine and the c-client library disagreed about how many messages 
there were, and presently this skew caused the crash (probably something 
like "Bad msgno 609 in mail_elt").

The attached patch to alpine/imap/src/c-client/imap4r1.c should protect 
against the bogus EXISTS.

However, in order to fully solve the problem, this bug needs to be fixed 
in Exchange pronto.  It is a bad one, and is likely to foul other IMAP 
clients.  Hopefully, our friends in Redmond are listening and will get 
someone to act on it soon.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
-------------- next part --------------
*** imap4r1.c	2007/06/05 21:39:37	1.23
--- imap4r1.c	2007/08/01 02:01:13
***************
*** 967,972 ****
--- 967,973 ----
        ambx.type = ASTRING;
        ambx.text = (void *) mb.mailbox;
        args[0] = &ambx; args[1] = NIL;
+       stream->nmsgs = 0;
        if (imap_OK (stream,reply = imap_send (stream,stream->rdonly ?
  					     "EXAMINE": "SELECT",args))) {
  	strcat (tmp,mb.mailbox);/* mailbox name */
***************
*** 3684,3691 ****
      t = strtok_r (NIL,"\n",&r);
  				/* now take the action */
  				/* change in size of mailbox */
!     if (!strcmp (s,"EXISTS")) mail_exists (stream,msgno);
!     else if (!strcmp (s,"RECENT")) mail_recent (stream,msgno);
      else if (!strcmp (s,"EXPUNGE") && msgno && (msgno <= stream->nmsgs)) {
        mailcache_t mc = (mailcache_t) mail_parameters (NIL,GET_CACHE,NIL);
        MESSAGECACHE *elt = (MESSAGECACHE *) (*mc) (stream,msgno,CH_ELT);
--- 3685,3694 ----
      t = strtok_r (NIL,"\n",&r);
  				/* now take the action */
  				/* change in size of mailbox */
!     if (!strcmp (s,"EXISTS") && (msgno >= stream->nmsgs))
!       mail_exists (stream,msgno);
!     else if (!strcmp (s,"RECENT") && (msgno <= stream->nmsgs))
!       mail_recent (stream,msgno);
      else if (!strcmp (s,"EXPUNGE") && msgno && (msgno <= stream->nmsgs)) {
        mailcache_t mc = (mailcache_t) mail_parameters (NIL,GET_CACHE,NIL);
        MESSAGECACHE *elt = (MESSAGECACHE *) (*mc) (stream,msgno,CH_ELT);
