MBOX-Line: From tss at iki.fi  Mon Mar 12 17:48:56 2007
To: imap-protocol@u.washington.edu
From: Timo Sirainen <tss@iki.fi>
Date: Fri Jun  8 12:34:38 2018
Subject: [Imap-protocol] QA / regression testing / stress testing IMAP
	pseudo-clients?
In-Reply-To: <alpine.OSX.0.83.0703111531070.20984@pangtzu.panda.com>
References: <"07Mar11. 134508pst. 57996"@synergy1.parc.xerox.com>
	<5aK4EfZGxwhHX3617xc/FQ.md5@libertango.oryx.com>
	<alpine.OSX.0.83.0703111531070.20984@pangtzu.panda.com>
Message-ID: <1173746936.9167.104.camel@hurina>

On Sun, 2007-03-11 at 15:50 -0700, Mark Crispin wrote:
> That's been the general problem all alone.  Another problem with test 
> suites is that it leads to servers which pass the test suite, but are 
> otherwise broken.  A server vendor should not have advance access to what 
> is in the test suite, but rather should be given a list of the tests that 
> the server failed.  Also, the tests should change over time (no memorizing 
> the exam!).

I think the test suite should do three things:

1. Verify that the server accepts everything that the ABNF rules say,
and that it doesn't reply anything that doesn't fit the ABNF rules.

2. Verify that the server in general works as the spec's text is
understood. Test all the commands as fully as possible.

3. Verify that the server doesn't do anything invalid when stress
testing with randomly (but correctly) behaving clients.

I'm most interested in implementing the 3rd part since it's the test
that can most easily expose server bugs. I was thinking about
implementing it something like:

First you get an mbox file containing whatever messages you want to test
(of course preferrably as complex as possible). Then you run a script
that uses UW-IMAP to access the mbox file and fetch all kinds of data
for all the messages (envelope, bodystructure, rfc822.size, body parts'
MD5 sums, etc.) All this information gets stored somewhere.

Then you run the actual imaptest tool that appends the mails from the
mbox file to the IMAP server, performs all the same fetch commands that
were used with UW-IMAP and makes sure that everything matches (allowing
some minor variations where they're legal).

So far this is pretty much the same as any other simple test that just
verifies everything to be correct. But I think the interesting part is
when you do all of this randomly with a lot of clients accessing the
mailbox simultaneously and each doing different things. The tester then
verifies that the state for all the connections is always consistent,
the IMAP protocol isn't violated and that replies contain all the
messages/data that is supposed to be there and that they're what they're
expected to be (eg. if FETCH reply is returning something as NIL or is
missing, do NOOP and make sure that EXPUNGE is received for that
message).

SEARCHes could also be checked by randomly generating different kinds of
SEARCH commands, feeding them to UW-IMAP and storing what messages they
match to. Then in the middle of the randomized testing make sure that
the SEARCH still matches those messages that exist in the mailbox at
that time.

\Recent flag's correctness could be tested by making the clients each
"own" those messages that are listed as \Recent for that session (doing
NOOPs while other clients are APPENDing new mails), making various
operations such as flag changes and expunges to them and always making
sure that no other client has done anything to the owned messages.

Once in a while there could be checkpoints where each connection does a
CHECK, then everyone waits until it's finished with all connections, and
then the connection states are compared to make sure everyone has the
exact same flags, and that STATUS command replies with the expected
counters.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/83f3b057/attachment.sig>
