MBOX-Line: From witold.krecicki at firma.o2.pl  Fri Jun 11 12:13:56 2010
To: imap-protocol@u.washington.edu
From: Witold =?utf-8?q?Kr=C4=99cicki?= <witold.krecicki@firma.o2.pl>
Date: Fri Jun  8 12:34:43 2018
Subject: [Imap-protocol] IMAP MOVE extension
In-Reply-To: <alpine.OSX.2.00.1006111111320.662@hsinghsing.panda.com>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>
	<201006111953.13490.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006111111320.662@hsinghsing.panda.com>
Message-ID: <201006112113.56572.witold.krecicki@firma.o2.pl>

On Friday 11 of June 2010 20:44:11 Mark Crispin wrote:
> On Fri, 11 Jun 2010, Witold Kr?cicki wrote:
> > You're saying that you can undo a failed COPY by deleting copied messages
> > and you can't undo MOVE by moving everything back. Situations in which
> > COPY using maildirs is not atomic:
> > 1. During COPYing filesystem error causes switch to read-only mode.
> > Several messages are copied, and cannot be deleted.
> > 2. During COPYing server crashes.
> 
> Yes, maildir has a problem complying with IMAP's atomicity requirements.
> Maildir is not a particularly good store for IMAP.
Yet it is the very most popular store for small-to-medium servers.
 
> > I am seeing Your point on how MOVE can fail to be atomic, but those are
> > exactly the same cases in which COPY can also fail to be atomic, so
> > where's the point?
> 
> The point is that it is a mistake to assume that maildir is a prototypical
> store for IMAP.  Any implementation using maildir is going to have
> problems.
>  
> MOVE doesn't solve those problems in maildir; it makes them worse.  MOVE
> doesn't solve those problems in other formats either.
No, MOVE keeps the problems in maildirs at the same level. It does not make 
them worse.

> >> Thus, the attractive idea of using rename for a maildir type store
> >> doesn't work.  You have to use something like hardlink+delete.  But
> >> even then you have error conditions.  What if, for some reason, the
> >> delete doesn't work?
> > 
> > Again, what if after copying half of the messages, for some reason,
> > delete doesn't work?
> 
> You're still thinking in terms of maildir as being a prototypical IMAP
> store.
And you're still thinking in terms of one and only implementation as the only 
correct one. 

> In an indexed store, the copied messages do not exist until the index is
> updated, and the index update is the final step and is atomic.  The worst
> consequence of a crash prior to this is unindexed data occupying
> inaccessible space that gets recovered in the next audit.  Some stores
> don't even have that problem.
> 
> MOVE provides no real benefit to such stores; yet those stores are the
> only ones that can actually implement it properly.
Database-backed stores can implement it properly and can have performance 
boost.

> >>> Also, the implementation of MOVE method in clients is, in most cases,
> >>> not a problem at all.
> >> 
> >> "[It is], in most cases, not a problem at all" is the earmark of
> >> inexperience and not having thought all these issues through.  Every
> >> separate code path for the same functionality adds 2^n complexity in
> >> overall code path.
> > 
> > It is already implemented in, at least, Thunderbird (for XAOL-MOVE).
> 
> Think about what happens when some server implements XAOL-MOVE in a way
> that is different than Thunderbird expects (since it has only ever seen an
> AOL server do it).
That would mean that the specification is not clear enough. As I can see this 
is pretty often in IMAP.

> >> MOVE is unimplementable as an atomic operation unless the DB works that
> >> way.
> > 
> > In any mail store in which mailbox membership is treated as 'mail label'
> > it is atomic.
> 
> And in such a case MOVE has no benefit.  There's no quota cost to COPY.
Yes, there is if there is only one 'mail label' per mail. Copying in this case 
is equivalent to inserting new record and, if handled inproperly, can be non-
atomic (comparing to fully atomic move)

> >> The first thing is to realize that rename can't work.  You have to use
> >> hardlink+delete.  And that's non-atomic and has danger points where an
> >> error can mean that you can't go forward or back.
> > 
> > In this way COPY is also non-atomic, as mentioned before.
> 
> That is one of many reasons why Maildir is not a particularly good store
> for IMAP.  Timo's server had to extend Maildir.
Again, yet it is the very most popular store for small-to-medium servers.

> >> Not easily.  It's possible with a database if the database is defined in
> >> a way to make it possible.  But in such databases it's a pointless
> >> tweak over pipelining.
> > 
> > In such databases it guarantees atomicity - after MOVE and dropped
> > connection client is SURE that message is either in source or
> > destination folder, and not in both at the same time.
> 
> What is the client use case?  What clients have expressed a need for move
> atomicity (which is the only reason not to use pipelining)?
You are pursuing the goal of all IMAP operations to be atomic, not me.  But 
read few lines above - I have written client case (specifically on weak 
connection clients such as GPRS/EDGE). 

> So far, you've been talking about server-based benefits, which I (and
> others) have shown are more illusory than real.  Adding MOVE will/can not
> fix a slow server implementation of COPY.
No. Only You have tried to show me that server-based benefits are illusory, and 
have shown no arguments to that except for examples from exisiting 
implementation.

> Consider, in particular, a commercial client product in which implementing
> this option costs real money and adds a support window of vulnerability
> (suppose the server advertises XAOL-MOVE but the server implementation is
> buggy?).
>
> These are REAL problems.  Right now, as we speak, the IMAP world faces
> client support issues because of servers that sends broken ENVELOPE and
> BODYSTRUCTURE data.  And, invariably, the answer is "it must be a bug in
> the client since it works fine with Outlook."
> 
> Multiple client code paths depending upon server is not a good thing.
Well, it sounds pretty weird hearing it from You. You have created a protocol 
that, by design, supported extensions. And the result is different capabilities 
of different implementations 
> >> John is a smart guy, but I'll bet that I can discover atomicity and/or
> >> error recovery flaws in AOL's move functionality.
> > 
> > UPDATE messages set folder='Trash' where uid in ... - where is the
> > atomicity flaw here (assuming that database is ACID)?
> 
> If AOL uses a database that permits that kind of operation, then there is
> no reason why COPY can not also work that way.
In this example COPY would create a need to insert new records, MOVE is just 
update. How could COPY work that way?
 
> There is no server benefit.  MOVE does not solve the problem of servers
> with slow COPY.  It does create the temptation for server implementors
> with slow COPY to create an unsafe, non-atomic, implementation of MOVE
> (e.g., using rename()).
So there is server benefit (as shown above)

> The only benefit is client benefit, and specifically that of an atomic
> operation.  But no use case has been given for it.
Read above.
 
> This is an "attractive nuisance".  It's one of those ideas that sounds
> really good on paper until you study the unintended consequences.
What consequences? 

> >> It's easy enough to say "that's alright, we don't care about these edge
> >> cases" in a private implementation.
> >> The problem is that it isn't alright, and you have to care, in a public
> >> specification.
> > 
> > You are assuming that it's impossible to implement this operation
> > properly (eg. atomic), and this is obviously false statement.
> 
> That sounds like inexperience on your part.
And that sounds like total inability to listen to your counterpart on Your 
part. 
I have gave you a good example of mail store (that I have seen) in which MOVE 
is atomic and in which MOVE is much faster than COPY - and you have ignored 
it. 
-- 
Witold Kr?cicki

Grupa o2 Sp??ka z o.o., ul. Jutrzenki 177, 02-231 Warszawa,
KRS 0000140518, S?d Rejonowy dla m.st. Warszawy, 
Kapita? zak?adowy 377.298,00 z?., NIP 521-31-11-513

