From: arw at ifu.net (arw)
Date: Fri, 9 Apr 1999 16:20:00 GMT
Subject: GadFly - MemoryError
Message-ID: <199904091623.MAA24288@ifu.ifu.net>
Content-Length: 2193
X-UID: 1629                                                  

My guess is that you have a lot of url matches and
gadfly is doing what you ask it to do.  Could be wrong.
Rewriting the query might help, as suggested elsewhere.
If you want to discuss this further I don't think we need
copy the list.  With 3000 entries you have a potential
9M matches.

Please try this

   select url, count(*) from table group by url order by 1 desc

And have a look at how many redundancies you have. This
query should run faster, I hope.

We could take this kind of discussion offline, if you like.  :)
   -- Aaron Watters

----Original Message-----
   >From:     	Oleg Broytmann <phd at sun.med.ru>
   >To:         	arw <arw at ifu.net>
   >Cc:         	Python Mailing List <python-list at cwi.nl>
   >Subject:     	RE: GadFly - MemoryError
   >Reply-To:     	phd at sun.med.ru
   >Date:    	Wednesday, April 07, 1999 4:16 AM
   >
   >Hello!
   >
   >On Mon, 5 Apr 1999, arw wrote:
   >> I don't know why you have this problem.
   >> Gadfly should not construct the cross product with
   >> the equality you mention present.
   >> (b1.URL = b2.URL)
   >
   >   I freed some memory and the program worked. It ate 30 Megs while running,
   >so I was in need of memory. But it ran 30 minutes (on 3000 rows!)
   >
   >> Please try adding an index on URL on the table
   >> in question.  Also, maybe try increasing your virtual
   >> memory (page file size, or whatever).
   >
   >   Indices or compiled kjbuckets was not of big help - instead of running
   >35 minutes the program ran 30. I think, my computer spent more time
   >swapping, not working.
   >
   >> Good luck.  fwiw, I've definitely run tables with
   >> 100k rows in Gadfly on a 64Meg machine.
   >>    -- Aaron Watters
   >
   >   You are in luck.
   >
   >> ps: let me know how it goes.
   >
   >   Not so good. I think I would not use gadfly, at least for this project
   >(currently I am using very simple "database", that stored in text files. Of
   >course, I cannot make complex requests, sucj as joins).
   >
   >Oleg.
   >---- 
   >    Oleg Broytmann  National Research Surgery Centre  http://sun.med.ru/~phd/
   >           Programmers don't die, they just GOSUB without RETURN.
   >
   >
   >





