81 lines
3.0 KiB
Plaintext
81 lines
3.0 KiB
Plaintext
From: stadt at cs.utwente.nl (Richard van de Stadt)
|
|
Date: Mon, 26 Apr 1999 14:29:58 +0200
|
|
Subject: Kosovo database; Python speed
|
|
References: <371DAAC2.D9046550@cs.utwente.nl> <371DB480.32947654@appliedbiometrics.com>
|
|
Message-ID: <37245C46.C13AAB62@cs.utwente.nl>
|
|
Content-Length: 2739
|
|
X-UID: 1451
|
|
|
|
Christian Tismer wrote:
|
|
>
|
|
> Richard van de Stadt wrote:
|
|
> >
|
|
> > Suppose we were going to make a database to help Kosovars locate
|
|
> > their family members. This would probably result in hundreds of
|
|
> > thousands of records (say 1 record (file) per person).
|
|
> >
|
|
> > Would Python be fast enough to manage this data, make queries on
|
|
> > the data, or should compiled programs be used?
|
|
>
|
|
> Dependant of the operating system, I'd suggest to use a
|
|
> database extension.
|
|
> Controlling this database from Python will give you
|
|
> enough speed. If I had to do this, my preferences are
|
|
>
|
|
> mySQL for Linux, with its interface,
|
|
> MS-Access for Windows, with a COM interface.
|
|
>
|
|
> The latter is not since I like it so much, but we have
|
|
> used it before, and the interfaces are there.
|
|
>
|
|
> Since the Kosovars need help quickly, I'd use this combination
|
|
> instead of writing something special.
|
|
|
|
I developed a system over the last few years which allows online
|
|
paper submission and retrieval, which we expect can quite easily
|
|
be transformed and reused to create a first prototype. On an old
|
|
system (SS10, 128MB RAM), Python is able to copy a test file
|
|
about 25000 times per minute, so I expect Python to be fast
|
|
enough, but wondered if other projects exist which also use
|
|
several 100.000's of records.
|
|
|
|
> Python alone will not
|
|
> be too easy, since your data will probably not fit into memory.
|
|
|
|
We were donated a system that is, I think, used for videoconferencing.
|
|
This probably is a Sun system, running Solaris, with Python 2.5.1
|
|
available. I expect at least .5 GB of RAM.
|
|
|
|
> You will also have lots of edits, so I think using a true
|
|
> database is the better choice here. (Not saying that Access is
|
|
> a true database, but it works fine with several 100000 records).
|
|
>
|
|
> But two single columns with a name and a record ID will fit,
|
|
> so your code might extract this info as a whole, map it to a dict
|
|
> and search it in some sophisticated manner. This can be even faster
|
|
> than the database.
|
|
> Do you have more info on the amount of data, fields per record,
|
|
> and what search capabilities are needed? Is it designed as a web
|
|
> based application? Are there on-line updates and such?
|
|
|
|
We intend to store any data that might be helpful, which includes photos.
|
|
Online submissions may not always be possible from within the camps,
|
|
but as refugees are being spread all over Europe, we think that it
|
|
could be used more often.
|
|
|
|
We'd like to collect existing databases, merge them, and provide all
|
|
kinds of name matching possibilities. Offline consulting and submission
|
|
should also be available, so probably there Access might then be used.
|
|
|
|
> ciao - chris
|
|
>
|
|
> --
|
|
> Christian Tismer :^) <mailto:tismer at appliedbiometrics.com>
|
|
[...]
|
|
|
|
Richard.
|
|
|
|
|
|
|
|
|