MBOX-Line: From witold.krecicki at firma.o2.pl  Fri Jun 11 10:53:13 2010
To: imap-protocol@u.washington.edu
From: Witold =?utf-8?q?Kr=C4=99cicki?= <witold.krecicki@firma.o2.pl>
Date: Fri Jun  8 12:34:43 2018
Subject: [Imap-protocol] IMAP MOVE extension
In-Reply-To: <alpine.OSX.2.00.1006110938590.662@hsinghsing.panda.com>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>
	<201006111817.39761.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006110938590.662@hsinghsing.panda.com>
Message-ID: <201006111953.13490.witold.krecicki@firma.o2.pl>

On Friday 11 of June 2010 19:06:00 Mark Crispin wrote:
> On Fri, 11 Jun 2010, Witold Kr?cicki wrote:
> >> Commands in IMAP are atomic; thus this MOVE operation must be atomic.
> >> It is simply impossible to do an atomic MOVE with many stores.
> > 
> > That's why it's an option, not a requirement.
> 
> Options are not a good thing
Looking at number of possible extensions to imap and number of ones supported 
by popular IMAP servers
> >> Note that not even a one-file/one-message store (ala maildir) can do
> >> MOVE atomically.  Each file has to be renamed separately.  That's not
> >> atomic!
> > 
> > In this way COPY is only atomic if destination mailbox is locked, so the
> > only problem with maildir store is locking two mailboxes simultanously
> 
> All implementations of IMAP must implement COPY as atomic.  That is a
> requirement of the protocol.
> 
> The problem is more than just locking.  The problem is also error
> recovery.
> 
> In a last resort, you can undo a failed COPY by deleting all
> the copied messages.  In the event of a failed MOVE, you have to move
> everything back.  But whatever caused the MOVE to fail can also cause the
> move back to fail.  Oh dear.  Oh my.
Do any of IMAP servers implement COPY as atomic?
You're saying that you can undo a failed COPY by deleting copied messages and 
you can't undo MOVE by moving everything back. Situations in which COPY using 
maildirs is not atomic:
1. During COPYing filesystem error causes switch to read-only mode. Several 
messages are copied, and cannot be deleted.
2. During COPYing server crashes.

I am seeing Your point on how MOVE can fail to be atomic, but those are 
exactly the same cases in which COPY can also fail to be atomic, so where's 
the point?

> Thus, the attractive idea of using rename for a maildir type store doesn't
> work.  You have to use something like hardlink+delete.  But even then you
> have error conditions.  What if, for some reason, the delete doesn't work?
Again, what if after copying half of the messages, for some reason, delete 
doesn't work?

> > Also, the implementation of MOVE method in clients is, in most cases, not
> > a problem at all.
> 
> "[It is], in most cases, not a problem at all" is the earmark of
> inexperience and not having thought all these issues through.  Every
> separate code path for the same functionality adds 2^n complexity in
> overall code path.
It is already implemented in, at least, Thunderbird (for XAOL-MOVE). 
 
> >> Note that in the one case where you can implement MOVE (see above),
> >> there is no particular benefit (other than the atomic lock) over
> >> pipelining because in the database the COPY operation is also an index
> >> operation.
> > 
> > Depends on DB design.
> 
> MOVE is unimplementable as an atomic operation unless the DB works that
> way.
In any mail store in which mailbox membership is treated as 'mail label' it is 
atomic. 

> > Maildirs can implement move (as an atomic operation) as long as locking
> > two mailboxes is permitted.
> 
> No they can not.  Atomicity is more than just locking.  There is also
> error recovery, and that is a MUCH harder problem.
>
> The first thing is to realize that rename can't work.  You have to use
> hardlink+delete.  And that's non-atomic and has danger points where an
> error can mean that you can't go forward or back.
In this way COPY is also non-atomic, as mentioned before.
 
> > Database-backed designs can implement atomic move easily.
> 
> Not easily.  It's possible with a database if the database is defined in a
> way to make it possible.  But in such databases it's a pointless tweak
> over pipelining.
In such databases it guarantees atomicity - after MOVE and dropped connection 
client is SURE that message is either in source or destination folder, and not 
in both at the same time.
 
> > the example of AOL (as mentioned by John Snow) shows that there is a need
> > for this functionality to be standarized so that clients that want to
> > use it won't need to check IMAP server domain name to figure out if it
> > can use eg. XAOL- MOVE.
> 
> John is a smart guy, but I'll bet that I can discover atomicity and/or
> error recovery flaws in AOL's move functionality.
UPDATE messages set folder='Trash' where uid in ... - where is the atomicity 
flaw here (assuming that database is ACID)?

> It's easy enough to say "that's alright, we don't care about these edge
> cases" in a private implementation.
>
> The problem is that it isn't alright, and you have to care, in a public
> specification.
You are assuming that it's impossible to implement this operation properly 
(eg. atomic), and this is obviously false statement.

-- 
Witold Kr?cicki

Grupa o2 Sp??ka z o.o., ul. Jutrzenki 177, 02-231 Warszawa,
KRS 0000140518, S?d Rejonowy dla m.st. Warszawy, 
Kapita? zak?adowy 377.298,00 z?., NIP 521-31-11-513

