MBOX-Line: From mrc+imap at panda.com  Tue Nov  1 00:00:12 2011
To: imap-protocol@u.washington.edu
From: Mark Crispin <mrc+imap@panda.com>
Date: Fri Jun  8 12:34:47 2018
Subject: [Imap-protocol] BODY.PEEK[section]<origin.size> FETCH response
In-Reply-To: <D96D7BAA-894A-41DC-8349-752E737C7A3B@iki.fi>
References: <8F0DA5FA-FB07-4AFA-9C58-8F0927998343@ghoti.org>
	<alpine.OSX.2.00.1110301823210.9034@hsinghsing.panda.com>
	<CABa8R6uyrHJ6AqQoGfMcLztG-zovaVK4FEFep8ibpYc2-6b7ig@mail.gmail.com>
	<alpine.OSX.2.00.1110311352000.9034@hsinghsing.panda.com>
	<CABa8R6ugaAADsrvaS1Nnau5bi7k9728b+x00NcG1Nbxfk9xwsA@mail.gmail.com>
	<alpine.OSX.2.00.1110311455120.9034@hsinghsing.panda.com>
	<CABa8R6vJGTa+VH-TOgXhCSpX5N1HZTBBqisDhPY9Metz5864JQ@mail.gmail.com>
	<D96D7BAA-894A-41DC-8349-752E737C7A3B@iki.fi>
Message-ID: <alpine.OSX.2.00.1110312320370.9034@hsinghsing.panda.com>

On Tue, 1 Nov 2011, Timo Sirainen wrote:
> A partial fetch
> from a non-zero offset requires some scanning to find out the
> LF-only-offset. But luckily all clients just fetch the blocks in
> increasing order from zero offset, so this isn't such an important
> problem.

In legacy stores that use LF-only newlines, I convert a body part to be
fetched to its CRLF form prior to output, and keep it in a buffer that is
reused when the fetched body part changes. When a message is parsed, both
the physical offset/size on disk and the IMAP count is calculated and
preserved for the duration of the session.

In the case of partial fetching, the conversion only happens once, since
as Timo notes all clients just fetch the blocks in increasing order from 0
offset. Each subsequent block fetch notices that the buffer already has
the desired body part, and thus no new mail store read/conversion is
needed.

Modern (post-1995) stores use CRLF newlines, or more accurately do no
newline conversion; they store the message exactly as it was transmitted
in SMTP; thus the physical size and IMAP count are the same. The CPU
saving in abolishing newline conversion is easily worth the storage cost.

Even more modern stores store all parse state, including offsets and
counts, in metadata so that once calculated it is never calculated again
(another great CPU savings).

Bad counts are a problem in partial fetching. The client needs correct
counts in order to form requests to fetch the right amount of data.
Otherwise, the client may fail to fetch the entire data (short count), or
get hung trying to fetch unavailable data (long count).

The check in Alpine that you refer to reports a long count. There's no way
for any IMAP client to detect a short count; it will just fail to fetch
the entire segment. So by advocating removing the check, you are asserting
that Gmail only does long counts, never short counts; and that all other
broken servers will be the same. Thus, you imply that users should not be
warned about servers whose developers are lazy and incompetent. And, when
the body part is truncated, you claim "it must be a client bug".

Most other server developers get it right. Why can't Google?

Perhaps we need product liability for software vendors after all.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

