MBOX-Line: From mrc+imap at panda.com  Fri Jun 11 11:44:11 2010
To: imap-protocol@u.washington.edu
From: Mark Crispin <mrc+imap@panda.com>
Date: Fri Jun  8 12:34:43 2018
Subject: [Imap-protocol] IMAP MOVE extension
In-Reply-To: <201006111953.13490.witold.krecicki@firma.o2.pl>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>
	<201006111817.39761.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006110938590.662@hsinghsing.panda.com>
	<201006111953.13490.witold.krecicki@firma.o2.pl>
Message-ID: <alpine.OSX.2.00.1006111111320.662@hsinghsing.panda.com>

On Fri, 11 Jun 2010, Witold Kr?cicki wrote:
>> Options are not a good thing
> Looking at number of possible extensions to imap and number of ones supported
> by popular IMAP servers

The vast majority of which were mistakes and should not have happened.

Yes, there are open source servers which implement every goofy extension
under the sun.  But when you look at the Real World and which extensions
have a business case, the set becomes very small indeed.

> Do any of IMAP servers implement COPY as atomic?

All compliant IMAP servers do.

> You're saying that you can undo a failed COPY by deleting copied messages and
> you can't undo MOVE by moving everything back. Situations in which COPY using
> maildirs is not atomic:
> 1. During COPYing filesystem error causes switch to read-only mode. Several
> messages are copied, and cannot be deleted.
> 2. During COPYing server crashes.

Yes, maildir has a problem complying with IMAP's atomicity requirements.
Maildir is not a particularly good store for IMAP.

> I am seeing Your point on how MOVE can fail to be atomic, but those are
> exactly the same cases in which COPY can also fail to be atomic, so where's
> the point?

The point is that it is a mistake to assume that maildir is a prototypical
store for IMAP.  Any implementation using maildir is going to have
problems.

MOVE doesn't solve those problems in maildir; it makes them worse.  MOVE
doesn't solve those problems in other formats either.

>> Thus, the attractive idea of using rename for a maildir type store doesn't
>> work.  You have to use something like hardlink+delete.  But even then you
>> have error conditions.  What if, for some reason, the delete doesn't work?
> Again, what if after copying half of the messages, for some reason, delete
> doesn't work?

You're still thinking in terms of maildir as being a prototypical IMAP
store.

In an indexed store, the copied messages do not exist until the index is
updated, and the index update is the final step and is atomic.  The worst
consequence of a crash prior to this is unindexed data occupying
inaccessible space that gets recovered in the next audit.  Some stores
don't even have that problem.

MOVE provides no real benefit to such stores; yet those stores are the
only ones that can actually implement it properly.

>>> Also, the implementation of MOVE method in clients is, in most cases, not
>>> a problem at all.
>> "[It is], in most cases, not a problem at all" is the earmark of
>> inexperience and not having thought all these issues through.  Every
>> separate code path for the same functionality adds 2^n complexity in
>> overall code path.
> It is already implemented in, at least, Thunderbird (for XAOL-MOVE).

Think about what happens when some server implements XAOL-MOVE in a way
that is different than Thunderbird expects (since it has only ever seen an
AOL server do it).

Experience says that this is a real (and quite expensive) problem.

>> MOVE is unimplementable as an atomic operation unless the DB works that
>> way.
> In any mail store in which mailbox membership is treated as 'mail label' it is
> atomic.

And in such a case MOVE has no benefit.  There's no quota cost to COPY.

>> The first thing is to realize that rename can't work.  You have to use
>> hardlink+delete.  And that's non-atomic and has danger points where an
>> error can mean that you can't go forward or back.
> In this way COPY is also non-atomic, as mentioned before.

That is one of many reasons why Maildir is not a particularly good store
for IMAP.  Timo's server had to extend Maildir.

>> Not easily.  It's possible with a database if the database is defined in a
>> way to make it possible.  But in such databases it's a pointless tweak
>> over pipelining.
> In such databases it guarantees atomicity - after MOVE and dropped connection
> client is SURE that message is either in source or destination folder, and not
> in both at the same time.

What is the client use case?  What clients have expressed a need for move
atomicity (which is the only reason not to use pipelining)?

So far, you've been talking about server-based benefits, which I (and
others) have shown are more illusory than real.  Adding MOVE will/can not
fix a slow server implementation of COPY.

Consider, in particular, a commercial client product in which implementing
this option costs real money and adds a support window of vulnerability
(suppose the server advertises XAOL-MOVE but the server implementation is
buggy?).

These are REAL problems.  Right now, as we speak, the IMAP world faces
client support issues because of servers that sends broken ENVELOPE and
BODYSTRUCTURE data.  And, invariably, the answer is "it must be a bug in
the client since it works fine with Outlook."

Multiple client code paths depending upon server is not a good thing.

>> John is a smart guy, but I'll bet that I can discover atomicity and/or
>> error recovery flaws in AOL's move functionality.
> UPDATE messages set folder='Trash' where uid in ... - where is the atomicity
> flaw here (assuming that database is ACID)?

If AOL uses a database that permits that kind of operation, then there is
no reason why COPY can not also work that way.

There is no server benefit.  MOVE does not solve the problem of servers
with slow COPY.  It does create the temptation for server implementors
with slow COPY to create an unsafe, non-atomic, implementation of MOVE
(e.g., using rename()).

The only benefit is client benefit, and specifically that of an atomic
operation.  But no use case has been given for it.

This is an "attractive nuisance".  It's one of those ideas that sounds
really good on paper until you study the unintended consequences.

>> It's easy enough to say "that's alright, we don't care about these edge
>> cases" in a private implementation.
>> The problem is that it isn't alright, and you have to care, in a public
>> specification.
> You are assuming that it's impossible to implement this operation properly
> (eg. atomic), and this is obviously false statement.

That sounds like inexperience on your part.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
