MBOX-Line: From snowjn at aol.com  Fri Jun 11 23:58:47 2010
To: imap-protocol@u.washington.edu
From: John Snow <snowjn@aol.com>
Date: Fri Jun  8 12:34:44 2018
Subject: [Imap-protocol] IMAP MOVE extension
In-Reply-To: <alpine.OSX.2.00.1006112111250.662@hsinghsing.panda.com>
References: <201006110854.37969.witold.krecicki@firma.o2.pl>	<201006112113.56572.witold.krecicki@firma.o2.pl>	<alpine.OSX.2.00.1006111214020.662@hsinghsing.panda.com>	<201006112352.05642.witold.krecicki@firma.o2.pl>
	<alpine.OSX.2.00.1006112111250.662@hsinghsing.panda.com>
Message-ID: <4C133027.1080603@aol.com>



Mark Crispin wrote:
> On Fri, 11 Jun 2010, Witold Kr?cicki wrote:
>>> IMAP has no guaranteed means of removing a message.  Study the 
>>> semantics
>>> of STORE and EXPUNGE carefully.  MOVE, on the other hand, requires that
>>> guarantee; and does not have the semantics of STORE and EXPUNGE.
>> I really don't get the point, please be more specific.
>
> Neither STORE, nor EXPUNGE, promise that they will store or expunge
> anything.  They do, or not do, whatever seems fit according to their
> discretion.  Generally, what they decide to more corresponds to what the
> user intended.  But there's no promises.
>
> Because of this, both STORE and EXPUNGE send back untagged responses
> stating what they actually did.  The client uses this to update its 
> cached
> state of what is on the server.  If it were not for these untagged
> responses, the client wouldn't have a clue as to what is going on.
>
> There are certain circumstances, such as when closing a mailbox in a
> client that does not cache offline state, that the client does not care
> what the server did beyond a vague presumption that that the server did
> more or less reasonably was what the user wanted.  For this reason, there
> STORE's .SILENT form, and the CLOSE command, exist.
>
> A successful MOVE, on the other hand, is an unconditional guarantee that
> all the requested message(s) were moved.  But it it?  Similarly, a failed
> MOVE ought to be an unconditional guarantee that none of the requested
> messages(s) were moved.  But, once again, is it?
>

Why does that have to be true?  If it's ok for store and expunge to send 
back untagged
responses and since you describe the move as another command that can 
expunge,
wouldn't the move result in untagged expunge responses?   I think it 
does. At least that's
how my implementation works.    In the event that some moves failed the 
client would
know where.

> What happens when there is an error in the middle of the move?  Is 
> there a
> state in which some messages were moved and others were not?  Is there a
> state in which some messages appear in both places (so it was a copy 
> after
> all)?  Is there a state in which some messages vanished completely?
>
> The only way that you can prevent all of these states is with a
> simultaneous commit in which a failure of either side (remove from 
> source,
> insert in destination) completely undoes everything.  Locking by itself
> isn't good enough; locking only prevents external races.  You still have
> a chicken-and-egg situation in which neither side can safely commit
> without confirmation that the other side has committed.
>
> There are ways to do it.  But these are not simple nor commonly
> understood.  If I talk about this stuff with most programmers their eyes
> glaze over.
>
> Not even the (much simpler) synchronization timing requirements of IMAP
> are well-implemented.  Timo's server and my Panda server are the only 
> ones
> that do it correctly in all circumstances (I'm hoping to add my new 
> server
> to the list).
>
> So let's say that we punt and say that an error is an error and we'll
> break the all-of-nothing guarantee of IMAP.
>
> There's now the problem of how what was actually done is communicated to
> the client.  The untagged responses from FETCH and EXPUNGE won't do it.
>
> So now you have to add a whole new framework of new untagged responses to
> your "simple" MOVE command to convey all the possible error conditions.
> And it's just not what happens in your server.  It's what can happen in
> any server.
>
> That is the problem with response to complex situations with simple
> solutions; it backfires.
>
>> COPY is by definition a slow operation. Using hardlinks etc. is a 
>> performance
>> hack, not a normal method.
>
> Nowhere in the definition of COPY does it say that it is a slow 
> operation.
> I should know.  I wrote it.
>
> Furthermore, your calling the use of hardlinks a "performance hack" is a
> personal insult to me.  I anticipated the use of hardlinks in IMAP from
> the very beginning, influenced by a pioneering mail store of the early
> 1980s.  I am designing a new store that carries that technique to a new
> level.  I already added stubbing to my 2006 store design last year; but
> the new design is far more advanced.
>
> Last but not least, it betrays a lack of understanding of how modern
> filesystems work.  Rename uses hardlinks internally when the target is on
> the system filesystem (when it isn't, it has to copy+delete).  Rename 
> is a
> world of hurt on a filesystem without hardlinks; and ultimately the
> problem is left for the disk auditor to resolve after a crash.
>
>> COPY creates NEW entity, MOVE only changes properties of an EXISTING 
>> entity.
>
> Now you are presupposing that the message itself knows where it is
> located.  That's fine if you are developing Gmail.  But all the world is
> not Gmail.
>
> Nor is that how modern filesystems work.  The inode has no clue where it
> is in the filesystem tree.
>
> The next step is the abolish the filesystem.
>
>> If COPY is implementable then MOVE is implementable. period.
>
> Wrong.  There are many mail stores in which MOVE is completely
> unimplementable.  The only thing that you can do is simulate the 
> effect of
> MOVE by using COPY then delete.
>
>> In Maildir it can (and will!) cause indeterminate results to client 
>> only in
>> case of catastrophic failure.
>
> Maildir has numerous failure modes short of catastrophic failure.  Timo
> can go into this in quite a bit of detail.
>
>> So will COPY in case of catastrophic failure.
>
> Not at all.  A mail store can be designed so that there is only a single
> operation: commit updated index.  It's much easier to do safe commit of a
> single object than two simultaneous objects.
>
>> MOVE creates advantage for any kind of single-labeled flat store.
>
> So all the world is all designed around Gmail?
>
>> Currently You are the only one more experienced that is trying to 
>> explain the
>> complexities of the situation. Most of the people here (I'm silently 
>> assuming
>> that they are not all newcomers) agreed that it is in fact a needed 
>> feature.
>
> Now you're talking like the cretins who changed the formerly predictable
> and deterministic behavior of the motion keys (e.g., CTRL/A, CTRL/E,
> CTRL/N, and CTRL/P) in emacs 23.  Apparently it was a "needed feature" to
> make emacs more like a word processor instead of a text editor.
>
> It is a typical "response to complex situations with simple solutions 
> that
> backfires."  The backfire here is that 34 years worth of scripts no 
> longer
> work correctly, with resulting subtle file corruption depending upon the
> window width.

I get the impression that your main objection to a move is that it may
be difficult or even impossible to implement depending on the storage 
system. Should
that really be considered when designing the protocol?  If you assume 
that it would be
easy for everyone to add, would the feature be a good idea or not?

I have already implemented a move command.  It was done as extension command
 to support our webmail client and several desktop clients.  We have 
found it to be
very useful and I do recommend that we find a way to add it.  I have  my
version for my clients, but I'd rather have an accepted standard that 
other clients
might choose to use it.


>
> -- Mark --
>
> http://panda.com/mrc
> Democracy is two wolves and a sheep deciding what to eat for lunch.
> Liberty is a well-armed sheep contesting the vote.
> ------------------------------------------------------------------------
>
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20100612/23e1a12c/attachment.html>
