MBOX-Line: From mrc+imap at panda.com  Fri Jun 11 20:15:20 2010
To: imap-protocol@u.washington.edu
From: Mark Crispin <mrc+imap@panda.com>
Date: Fri Jun  8 12:34:44 2018
Subject: [Imap-protocol] Re: IMAP MOVE extension
In-Reply-To: <20100612015450.GA4440@dan.olp.net>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>
	<201006120159.05366.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006111725170.662@hsinghsing.panda.com>
	<201006120251.24779.witold.krecicki@firma.o2.pl>
	<20100612015450.GA4440@dan.olp.net>
Message-ID: <alpine.OSX.2.00.1006111924370.662@hsinghsing.panda.com>

On Fri, 11 Jun 2010, Dan White wrote:
> I would suggest focusing your arguments on the benefit of the change to the
> protocol itself.

The demerits of the proposal:

[1] It adds complexity to any implementation that uses it.  In the history
of IMAP, other extensions that solved far more serious problems ended up
with little implementation.

[2] This will be the first command other than EXPUNGE that expunges a
message.

[3] It can not be implemented on many stores as anything better than an
internal COPY + UID EXPUNGE.  The disk quota and performance benefits do
not happen in this case.  Failure is indeterminate because the failure
could happen in the second step, possibly resulting in messages staying in
the source mailbox (which violates the claimed semantics of MOVE).  This
is what killed MOVE in 1986.

[4] With Maildir class stores:
     [a] One obvious (wrong) implementation is to use rename.  This creates
 	an indeterminate failure in which some, but not all, messages were
 	moved.  In addition, if the source and destination are on
 	different filesystems, then rename actually degenerates to [3].
     [b] A better but still wrong implementation is to use hardlink then
 	expunge from source.  This is actually just a form of [3], but
 	with better performance for COPY.  It still has the same
 	indeterminate failure.  Also, if the source and destination are
 	on different filesystems, you can't hardlink.

[5] With indexed stores, including Maildir extended with an index (the
filesystem directory isn't really good enough to be an index), move can be
properly implemented IF and ONLY IF it is possible to commit the update to
both indices simultaneously and the commit operation is guaranteed atomic
all-success or all-fail.  Otherwise, once again, you end up with an
indeterminate failure.

There are further complications if, as in Maildir, you actually have to
move the message data to another directory instead of just writing updated
pointers in the indices.  It's much easier with a CAstor type storage
layout, especially since you can just drop file pointers that you no
longer want (CAstor abolishes the filesystem - whoo hoo!).


In conclusion:

The underlying idea of MOVE is (as it has been since the advent of
Maildir) to "just do rename" with a Maildir class store.  It's a tempting
idea, but this is a temptation that needs to be resisted.  It's all fine
and good to talk about databases, but speaking as someone who actually
does it (including without a filesystem, which is a truly wonderful
solution for many problems), there aren't easy solutions there.

Rename is traditionally a nasty operation in any data store.  It is
invariably the most fragile and buggy user operation in a filesystem.
Rename has semantic problems as well; the IMAP RENAME command is
totally broken in its design.  No server implements it correctly.  If I
had dictatorial power I would have abolished RENAME from the specification
long ago.

A correct implementation of MOVE must either place ALL requested messages
in the target and remove them from the source, OR do nothing.  There is no
way in the framework of IMAP for a MOVE command to say:
 	"some messages were moved, but not all"
  or	"some (or all) messages are duplicated in both"
  or	"some messages vanished into thin air"

All of these are possible unless there is a simultaneous all-or-none
commit on both sides.  The only one of these three errors that you can
abolish is the third; by not using rename and by putting yourself at
greater risk of the first two.

Put another way, any error, including a commit error, on either end has to
stop the commit on the other end.  It's a chicken-and-egg problem; neither
end can safely commit until the other end has successfully committed.  So
both indices have to be part of the same commit.  That's doable, but it's
not something that normal DBs do.  But if you don't do it, then you have a
timing race and one of the error states in the previous paragraph (none of
which have an expression within IMAP) occur.

The safest operation is a copy (which can be done by hardlink if the
filesystem permits) followed by source expunge.  But that isn't a MOVE;
that's a faster COPY.  And nothing prevents the implementation from doing
a faster COPY.

COPY does not have this problem.  COPY succeeds or fails when the commit
to check the messages in to the destination succeeds or fails.  If the
commit fails, then either COPY truncates away what it copied or it just
drops the pointers and lets whatever underlying cleanup of ownerless data
recover the space (if you care; in CAstor you don't).

I fear that if MOVE advances, it WILL be implemented by means of rename in
Maildir type stores, with the result that clients WILL encounter
surprising results.  If the server implementor isn't damn careful, those
surprises will include vanished messages.  A Maildir implementor with IMAP
has enough problems (Timo can elaborate on this somewhat) avoiding
messages vanishing as it is.

A further complication is with multiple simultaneous access to the same
mailbox (either source or destination).  It is easy enough to handwave and
say "use locks".  How many people have actually programmed complex
databases with simultaneous access and locks?

According to Timo's IMAP test results page, he and I are the only ones who
have ever gotten it right in IMAP; and that was only because he and I
played a jolly game of one-upsmanship with each other that ended in a (I
hope) mutually satisfying tie.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

