MBOX-Line: From tss at iki.fi  Sat Jun 12 07:58:17 2010
To: imap-protocol@u.washington.edu
From: Timo Sirainen <tss@iki.fi>
Date: Fri Jun  8 12:34:44 2018
Subject: [Imap-protocol] IMAP MOVE extension
In-Reply-To: <201006120958.13649.witold.krecicki@firma.o2.pl>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>
	<201006112352.05642.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006112111250.662@hsinghsing.panda.com>
	<201006120958.13649.witold.krecicki@firma.o2.pl>
Message-ID: <849822B6-69A5-4FEF-8865-BF8DD0D01957@iki.fi>

On 12.6.2010, at 8.58, Witold Kr?cicki wrote:

>> So let's say that we punt and say that an error is an error and we'll
>> break the all-of-nothing guarantee of IMAP.
> This guarantee is a myth that is not working in eg. Maildirs. 
> The same problems that appear with COPY or EXPUNGE in those critical 
> situations but nobody is thinking about removing COPY or EXPUNGE from the 
> protocol.

Since you keep talking about maildirs..:

Assumptions: Maildir can be accessed mostly without locks. Reading mails should never get any locks, and that's also how Dovecot works. Changing flags or expunging messages would be possible to do without locks, although currently Dovecot doesn't. Assigning UIDs to new messages (i.e. making new messages visible) pretty much requires some kind of locking.

The way I've implemented COPY is that it first link()s files to the destination Maildir/tmp/ directory. Then it locks the maildir, assigns UIDs to messages, rename()s files from tmp/ to new/ and unlocks. Messages won't be visible to other connections until unlock is complete. This means that as long as the system doesn't crash, there is a single atomic event (unlock) when the messages either become visible or the copy operation gets rollbacked. This can fail only if there are non-Dovecot MUAs accessing the maildir that aren't using Dovecot's locking, but in normal Dovecot-only situations this works perfectly.

Now, if you wanted to implement MOVE operation using rename(), there is a problem: Immediately after you do a rename() the message is gone from the source mailbox. If another session tries to open the mail (remember, no locks while reading), or for some other reason scans the maildir for what messages it contains, it sees that the message is gone and issues EXPUNGE to client. There is no way to revert that! And why would the MOVE operation fail? 1) COPY fails if one of the messages has already been expunged, so MOVE should probably too, and this won't be known beforehand. 2) Filesystem quota can cause rename() to fail if it would increase the destination directory's size.

So the difference is that COPY can under normal operating conditions work as intended, while MOVE implemented with rename() can't.

> What if the server deletes 3 out of 10 messages and then is struck by a 
> filesystem failure (cannot delete remaining 7, cannot undelete of the first 3)? 
> This situation is (formally) not covered in the protocol, server cannot send 
> untagged EXPUNGE responses for those 3 messages (because then it would have to 
> send OK) and it cannot send OK because EXPUNGE after a success removes ALL 
> messages with \Deleted flags...

I don't think that's a correct assumption.

> It should send NO but then the client would 
> think that no messages were deleted permanently...

I don't think that's a correct assumption either.

My server replies to EXPUNGE always with OK, even when there are \Deleted messages that weren't expunged because ACLs didn't allow that (it used to send NO, but some clients tried to send EXPUNGE internally and kept giving popup windows saying the EXPUNGE failed). Clients should use the untagged EXPUNGE replies to figure out what happened, the OK reply from EXPUNGE doesn't tell anything.
