From: stadt at cs.utwente.nl (Richard van de Stadt)
Date: Mon, 26 Apr 1999 14:29:58 +0200
Subject: Kosovo database; Python speed
References: <371DAAC2.D9046550@cs.utwente.nl> <371DB480.32947654@appliedbiometrics.com>
Message-ID: <37245C46.C13AAB62@cs.utwente.nl>
Content-Length: 2739
X-UID: 1451                                                  

Christian Tismer wrote:
> 
> Richard van de Stadt wrote:
> >
> > Suppose we were going to make a database to help Kosovars locate
> > their family members. This would probably result in hundreds of
> > thousands of records (say 1 record (file) per person).
> >
> > Would Python be fast enough to manage this data, make queries on
> > the data, or should compiled programs be used?
> 
> Dependant of the operating system, I'd suggest to use a
> database extension.
> Controlling this database from Python will give you
> enough speed. If I had to do this, my preferences are
> 
> mySQL for Linux, with its interface,
> MS-Access for Windows, with a COM interface.
> 
> The latter is not since I like it so much, but we have
> used it before, and the interfaces are there.
> 
> Since the Kosovars need help quickly, I'd use this combination
> instead of writing something special.

I developed a system over the last few years which allows online
paper submission and retrieval, which we expect can quite easily
be transformed and reused to create a first prototype. On an old
system (SS10, 128MB RAM), Python is able to copy a test file
about 25000 times per minute, so I expect Python to be fast
enough, but wondered if other projects exist which also use
several 100.000's of records.

> Python alone will not
> be too easy, since your data will probably not fit into memory.

We were donated a system that is, I think, used for videoconferencing.
This probably is a Sun system, running Solaris, with Python 2.5.1
available. I expect at least .5 GB of RAM.
 
> You will also have lots of edits, so I think using a true
> database is the better choice here. (Not saying that Access is
> a true database, but it works fine with several 100000 records).
> 
> But two single columns with a name and a record ID will fit,
> so your code might extract this info as a whole, map it to a dict
> and search it in some sophisticated manner. This can be even faster
> than the database.
> Do you have more info on the amount of data, fields per record,
> and what search capabilities are needed? Is it designed as a web
> based application? Are there on-line updates and such?

We intend to store any data that might be helpful, which includes photos.
Online submissions may not always be possible from within the camps,
but as refugees are being spread all over Europe, we think that it
could be used more often.

We'd like to collect existing databases, merge them, and provide all
kinds of name matching possibilities. Offline consulting and submission
should also be available, so probably there Access might then be used. 

> ciao - chris
> 
> --
> Christian Tismer             :^)   <mailto:tismer at appliedbiometrics.com>
[...]

Richard.




