82 lines
4.0 KiB
Plaintext
82 lines
4.0 KiB
Plaintext
MBOX-Line: From tss at iki.fi Mon Mar 12 17:48:56 2007
|
|
To: imap-protocol@u.washington.edu
|
|
From: Timo Sirainen <tss@iki.fi>
|
|
Date: Fri Jun 8 12:34:38 2018
|
|
Subject: [Imap-protocol] QA / regression testing / stress testing IMAP
|
|
pseudo-clients?
|
|
In-Reply-To: <alpine.OSX.0.83.0703111531070.20984@pangtzu.panda.com>
|
|
References: <"07Mar11. 134508pst. 57996"@synergy1.parc.xerox.com>
|
|
<5aK4EfZGxwhHX3617xc/FQ.md5@libertango.oryx.com>
|
|
<alpine.OSX.0.83.0703111531070.20984@pangtzu.panda.com>
|
|
Message-ID: <1173746936.9167.104.camel@hurina>
|
|
|
|
On Sun, 2007-03-11 at 15:50 -0700, Mark Crispin wrote:
|
|
> That's been the general problem all alone. Another problem with test
|
|
> suites is that it leads to servers which pass the test suite, but are
|
|
> otherwise broken. A server vendor should not have advance access to what
|
|
> is in the test suite, but rather should be given a list of the tests that
|
|
> the server failed. Also, the tests should change over time (no memorizing
|
|
> the exam!).
|
|
|
|
I think the test suite should do three things:
|
|
|
|
1. Verify that the server accepts everything that the ABNF rules say,
|
|
and that it doesn't reply anything that doesn't fit the ABNF rules.
|
|
|
|
2. Verify that the server in general works as the spec's text is
|
|
understood. Test all the commands as fully as possible.
|
|
|
|
3. Verify that the server doesn't do anything invalid when stress
|
|
testing with randomly (but correctly) behaving clients.
|
|
|
|
I'm most interested in implementing the 3rd part since it's the test
|
|
that can most easily expose server bugs. I was thinking about
|
|
implementing it something like:
|
|
|
|
First you get an mbox file containing whatever messages you want to test
|
|
(of course preferrably as complex as possible). Then you run a script
|
|
that uses UW-IMAP to access the mbox file and fetch all kinds of data
|
|
for all the messages (envelope, bodystructure, rfc822.size, body parts'
|
|
MD5 sums, etc.) All this information gets stored somewhere.
|
|
|
|
Then you run the actual imaptest tool that appends the mails from the
|
|
mbox file to the IMAP server, performs all the same fetch commands that
|
|
were used with UW-IMAP and makes sure that everything matches (allowing
|
|
some minor variations where they're legal).
|
|
|
|
So far this is pretty much the same as any other simple test that just
|
|
verifies everything to be correct. But I think the interesting part is
|
|
when you do all of this randomly with a lot of clients accessing the
|
|
mailbox simultaneously and each doing different things. The tester then
|
|
verifies that the state for all the connections is always consistent,
|
|
the IMAP protocol isn't violated and that replies contain all the
|
|
messages/data that is supposed to be there and that they're what they're
|
|
expected to be (eg. if FETCH reply is returning something as NIL or is
|
|
missing, do NOOP and make sure that EXPUNGE is received for that
|
|
message).
|
|
|
|
SEARCHes could also be checked by randomly generating different kinds of
|
|
SEARCH commands, feeding them to UW-IMAP and storing what messages they
|
|
match to. Then in the middle of the randomized testing make sure that
|
|
the SEARCH still matches those messages that exist in the mailbox at
|
|
that time.
|
|
|
|
\Recent flag's correctness could be tested by making the clients each
|
|
"own" those messages that are listed as \Recent for that session (doing
|
|
NOOPs while other clients are APPENDing new mails), making various
|
|
operations such as flag changes and expunges to them and always making
|
|
sure that no other client has done anything to the owned messages.
|
|
|
|
Once in a while there could be checkpoints where each connection does a
|
|
CHECK, then everyone waits until it's finished with all connections, and
|
|
then the connection states are compared to make sure everyone has the
|
|
exact same flags, and that STATUS command replies with the expected
|
|
counters.
|
|
-------------- next part --------------
|
|
A non-text attachment was scrubbed...
|
|
Name: signature.asc
|
|
Type: application/pgp-signature
|
|
Size: 196 bytes
|
|
Desc: This is a digitally signed message part
|
|
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/83f3b057/attachment.sig>
|