From: tismer at appliedbiometrics.com (Christian Tismer)
Date: Mon, 19 Apr 1999 21:06:06 GMT
Subject: Memory and swapping question
References: <371B5ED8.A9C82170@appliedbiometrics.com> <7fg1ep$t5s$1@nnrp1.dejanews.com>
Message-ID: <371B9ABE.3D2D8543@appliedbiometrics.com>
Content-Length: 2517
X-UID: 67                                                  


aaron_watters at my-dejanews.com wrote:
> 
> In article <371B5ED8.A9C82170 at appliedbiometrics.com>,
>   Christian Tismer <tismer at appliedbiometrics.com> wrote:
> > due to a question which came up in the tutor list, I'd like
> > to ask if somebody can explain the following:....
> 
> <chomp><chomp>timings on making huge lists of integers...
> 
> > On my system, creation takes about 10 times as for big/2,
> > this is ok. But the del takes at least three times as long.
> > Besides the fact that integers are never really disposed but
> > build up a freelist, why is deletion so much slower now?
> 
> Could be wrong, but this may be a case of a famous database
> problem.  The OS (typically) swaps out pages by picking the "least recently
> used" page, but when you are decreffing (scanning) a HUGE list of sequentially
> allocated objects this guarantees that the page you need next will
> be swapped out by the time you get to it. Yikes! Allocation is faster
> because you are really only "paging things out" (the first time)
> and the write to the disk can be buffered until the disk is
> ready, allowing the program to proceed (?I think?).

Exactly.
In this case, things are even worse:
Iin the de-allocation phase, the internal integer cache
is scanned in order, to return the integers to the
freelist. This treats a stack-like structure as a heap,
making integer deallocation like a list.reverse() on
the cache file.
It also applies to other objects which are created
in-order and referenced by the list. The same disk trashing.
Which means that the malloc routines use a similar,
stack-like freelist, at least on Win98.

> This is one reason why Oracle and Sybase, etc, like to do their own
> memory and disk management ("gimme them sectors -- don't need no
> g.d. filesystem, thanks!").  Just a guess, but a not completely
> uneducated one.

Well, I changed the deallocation strategy of lists to free
objects beginning from the end. (1 line in listobject.c)
Now, all my examples run as expected, with del no longer
being more expensive than create.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101   :    *Starship* http://starship.python.net
10553 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     we're tired of banana software - shipped green, ripens at home




