20060830 Wednesday August 30, 2006

Rapid Evaluation Madness

I‘ve wrapped up the last project using Ruby and ActiveRecord to extract big complex records about people and turn them into Wiki pages. Thanks to WikiMedia‘s XML import ../../page/format__the_process_wasn__8216.css;t too difficult. While I learned more about Wiki markup than I really cared to, the real story with that was how to pull 12,688 records that each had no less than 16 associated sets of records and attachments using ActiveRecord and Microsoft SQL Server. Once I got the connection working (using ADO) I quickly ran into a problem of scale. The way ActiveRecord works is that it uses lazy initialization to build all the objects as you need them. With a set of 12,688 complex objects (and associated attachments) I would run out of memory after processing around 1,000 records — and I have 2GB on my machine.

The way around the high volume of data problem is actually pretty simple. ActiveRecord really wasn‘t designed for this type of problem, so I had to work around a couple of limitations. First, I can‘t have 12,688 complex objects in memory at the same time (some of those attachments were about 25MB and that data was loaded in as well). I can have 12,688 simple IDs in an array. So I used a simple query to get the IDs and then looped through looking up each person by itself. Worked like a charm.

Now I‘m on a rapid application evaluation/development project to incorporate a survey management tool into the government infrastructure. I have to slog through some documentation before I get down and dirty. In the process of setting up the server I ran into a particularly ugly snag. The SELinux extensions prevent the MySQL server RPMS from running. I‘ve seen a couple links, but I don‘t have the authority to download the patches I need. This is quite frustrating, but as the high level architecture documentation takes precidence I have to push it off a bit. My due date is tomorrow. Yes, government hoops are fun. You spend a lot of time creating paper for people to skim through and never look at again. Such is life as a contractor.

(2006-08-30 19:27:23.0) Permalink Trackback


Comments are closed for this entry.