20060126 Thunderbird Inbox blank problem
Thursday January 26, 2006
Category: General
I managed to convince a client to use Firefox, Thunderbird and OpenOffice 2.0 in their 10-men office. A small victory for the open source solutions world. :) So far their experience has been ok, with only a handful of users complaining about certain additional steps needed in OpenOffice 2.0 when compared to MS Office to perform the same task.

However, this week a problem that I have not encountered before occurred: the director's Thunderbird on her notebook refuses to work! Basically, the symptoms are as follows:
  • When Thunderbird is opened, Inbox is displayed as blank, with the message "Building summary file" in the status bar
  • The memory usage of Thunderbird grows to a very big value.
  • The CPU Usage soars.
  • Other applications cannot response with the usual speed.

After a very long time building the summary file, the Inbox is displayed with around 1000+ files, but once I clicked on another folder and click back to Inbox, the same long building started again. It puzzled me because I remember also having large folders with more than 8000 emails, and I never encountered that problem before. I first thought it was a virus/spyware problem, but after scanning for everything, the machine was reported as clean from such problems. I even upgraded her Thunderbird client to version 1.5, but still the same problem appeared.

Took me a while with different search strings at google to finally find the mozillazine article on Inbox stays blank. Apparently the Inbox file was corrupted, and Thunderbird was not able to complete the building of the summary file properly. And the reason I did not encounter the problem was because I have more than 8000 emails in a folder but my folders are never larger than 50MB, whereas her single Inbox file was more than 200MB.

I followed the article until the point where it says "if this succeeds, you can move .....", and the next point at that time was "if this also fails, you may have to take your loss". The problem is when I tried to import the Inbox file the MboxImport extension reports that the file is not a valid mbox format file.

No way for my client to lose her 1000+ emails though, so I have to find another way to solve the problem. Knowing that the mbox file format is just a plain text, I proceeded to open it with Vim (which resulted in my other problem with Vim), and finally was able to see that the first line was corrupted and somehow became a very long time. I removed the first line and corrected the From statement (looking at other emails down the file and copied the From statement). I repeated the import using MboxImport and luckily now the file was recognised as an mbox file!

After being about to recover the emails in the one big Inbox, I proceeded to compact folders and moved older mails into separate folders, with the aim to keep individual mbox files small. Problem was solved, but I believe it will recurr some time in the future if the user is not educated to perform housekeeping regularly.

I updated the mozillazine article with 2 additional points to summarize my solution, hoping it would help someone else with a similar problem. However, it is quite puzzling that Thunderbird does not suggest a limit on the mailbox size so that users will know that keeping all emails in one folder is going to be hazardous to their "informational health".

Lesson learnt: (1) Compact your Thunderbird folders often. (2) Split up mails into folders to avoid single large mbox files. Maybe when I figure out how to submit enhancements/bugs to Thunderbird, I will submit a suggestion for Thunderbird to remind users to archive and compact emails (MS Outlook has such a reminder, but I almost never say yes to "Do you want to archive your emails" though. :P).
Posted on (2006-01-25 23:12:52.0) Permalink Comments [4]
20060126 Memory problem with VIM
Thursday January 26, 2006
Category: General
I love Vim, and I use it for all of my text editing until Eclipse came along. Now I program in Eclipse but still do a large portion of my text editing in Vim. Recently I was troubleshooting a problem with Thunderbird email, and had to open an Inbox file that was more than 200MB in size. With the past experiences with Vim opening large files, I did not hesitate before opening the Inbox file with Vim.

It took a long time (try more than 20 minutes) for Vim to load the whole file into memory, although I have 1GB of physical RAM on my notebook, but at least I was able to load the file and fix the problem. One thing that I observed is that the Mem Usage was 200MB, but the VM Size was around 400MB, double the Mem Usage.

Subsequently, when I tried to open another text file with Vim, the surprising fact is that my task manager still reports that Vim is trying to take up more than 400MB of memory to load the small 5KB text file!

A little research reveals that the _viminfo file stores the register for the last edit, and my last edit was the first line of the corrupted Inbox file, which I suspect to be a very long line. My _viminfo file was now also 400MB+ and every time I try to load Vim, it tries to load the registers into memory before allowing me to edit the text file.

Next logical thing for me to do was to either to delete the _viminfo file (which I think will be regenerated by Vim), or to open the _viminfo file to edit out the one long long line. With some time on my hands, I tried the second approach, just to see what happens.

It turned out that this was quite a bad idea. Now the file loads with more than 400MB under Mem Usage, and the VM Size climbed for more than 40 mins to around 1.3GB!! And the anti-climax was that at the end of 40mins, Vim loaded with the message that my machine has run out of memory. However, somehow Vim still loaded the _viminfo file, but I guess it has truncated a large portion of the file. I tried looking for the long line and it is not there, so I saved _viminfo, and noted that the size dropped to 2KB.

Next surprise is that when I closed Vim this time round, I was expecting everything to go back to normal. I was wrong because Vim still took a long time to close, and _viminfo went back to 400MB! Apparently after I saved _viminfo with the 2KB size, the application wrote back whatever it stored in memory and overwrote my changes.

Enough with the experimenting, I deleted _viminfo file and opened the small text file. Vim loaded with its usual fast speed, and a new (small) _viminfo file was generated when I closed Vim.

Lesson learnt: (1) If I opened a large file with a long line in Vim, I need to check the size of _viminfo before trying to open any other text files. (2) Never be so stupid as to try to open _viminfo with vim when _viminfo is very large. :)
Posted on (2006-01-25 22:19:20.0) Permalink Comments [0]
20060118 IE Z-Index bug
Wednesday January 18, 2006
Category: General
Was working with an problem with drop-down menus not covering the elements they should be covering. The same code worked in Firefox, but once I render the same page in IE, the bug appears. After much trial and error, I finally stumbled upon this article, which suggests that I should remove unneccesary position:relative elements. Solved my problem immediately!
Posted on (2006-01-18 04:28:17.0) Permalink Comments [0]
20060111 Hack to enable InspectElement in Firefox 1.5
Wednesday January 11, 2006
Category: General
After upgrading to Firefox 1.5, everything was fine, except that one of my favourite extensions - InspectElement - ceased to work. Luckily someone tried and came up with a hack to enable it in Firefox 1.5, just by simply changing the maxVersion number in the rdf file. See details of the hack at the InspectElement page. I tried it and it worked perfectly.
Posted on (2006-01-11 01:55:00.0) Permalink Comments [0]
20051022 GMail background upload of attachments
Saturday October 22, 2005
Category: General
Some time ago Gmail launched the new feature of automatically saving a draft of the email you are typing. Works very well and helped me once when my hand was too quick to click on the 'X' to close the browser, before my mind told me I was closing the wrong browser window.

Lately I have been uploading a fair amount of attachments via Gmail. On one occasion, I selected a file to be uploaded, and was distracted by something else and only came back to the email a few minutes later. Apparently in the mean time when I was busy with something else, the attachement was uploaded and cached somewhere even before I have complete typing the email.

After this, I notice that whenever a file attachment have been chosen by clicking on the "Browse" button, a small red box (probably 2x2 pixels) will appear on the top right hand corner of my Firefox browser, and the status bar will inform me that the browser is communicating with some site in google.

With this knowledge, a small tip for using Gmail is to select your file attachements before typing the email body. In this way, by the time you finish typing your email, the file attachment could already be uploaded completely and it will only take seconds to send out the email. Compare this with the typical behaviour of uploading the file only when the "Send" button is clicked, and you will find that this is a bit time-saver when you are talking about large file attachments.
Posted on (2005-10-21 14:27:33.0) Permalink Comments [1]
20051021 Moving projects into a new workspace
Friday October 21, 2005
Category: Eclipse
I was transitioning between Eclipse 3.0.1 and Eclipse 3.1 for a few weeks now, testing the features in Eclipse 3.1 to see if they are compatible with my projects created in Eclipse 3.0.1. Glad to say that so far I have not encountered any problems with Eclipse 3.1. Workspaces created using E3.0.1 can be opened with E3.1 with no issue.

This morning I decided to complete the migration to E3.1 so that I can free up some free space on my hdd, but found that previously I had created a workspace in the \workspace folder under e3.0.1 to store some of my other smaller projects. That was before I started the practice of creating a new workspace for each set of related projects and keeping them in a folder separate from the Eclipse installation. So I had to move these projects to a new workspace before removing the installation of e3.0.1.

I created a new workspace and tried to import ../../../page/dunpanic/projects_using_File-_gt.css;Import. Since I have existing projects, I selected "Existing Projects Into Workspace". The following dialog box allowed me to choose the root folder and the specific project to import. I tried one project, and it was imported properly into the new workspace - with one gotcha: the project files are still residing in the old folder, the new workspace only points to the old folder, allowing me to access the files from the e31 workspace. This is not what I want. What I want is to copy the projects physically to the folder where the new workspace is created.

I noticed that there is another option at the "Import Projects" dialog box for "Select archive file", so I tried zipping up my project in the old workspace and import using the .zip file into the new workspace. It works, with another glitch - somehow some of the subversion meta-data files (under the respective .svn folders) were not zipped up or unzipped properly, and the project is not recognised as an svn project.

Last approach I tried, which is the one that is working for me now:
  1. create an empty project in the new workspace.
  2. Copy the project using the filesystem from the old workspace to the empty folder in the new workspace, over-writing the .classpath and .project files created by creating the new project.
  3. refresh the project in eclipse

There must be a better way to do this, but since it is working for me, I am too lazy to explore the other approaches. [After note: one day after posting the above, I rethink the post, and thought to myself: why did I not think of the approach to simply copy the workspace folder from under E3.0.1 to another location and open the workspace with E3.1. That could be the most time efficient way of solving my problem, but since I have deleted the E3.0.1 installation, there is no way I can verify whether this works.]
Posted on (2005-10-20 22:39:58.0) Permalink Comments [0]
20051009 Adding Spring to my webapp
Sunday October 09, 2005
Category: Spring
Have started to learn about Spring, and am excited about integrating it with my current projects. Today I started making small changes to some of the classes to use the Spring Core package, more for the IoC features. My objective is to make my changes as unobtrusive as possible, and to affected only the few classes that I am changing.

Just to note down the few changes I have made.

1. First, I added the dependency for the Spring Framework in my maven 1 project.xml:
2. After this, I added the my context definition file applicationContext-main to the folder under which I store my util classes at src/java/com/mydomain/util. The application context definition can be anyway, as long as it can be found by the context loader later.

3. All my JUnit test classes extend from a BaseTestCase, so I added the following code block in BaseTestCase:
    protected final static ApplicationContext ctx;
    // This static block ensures that Spring's BeanFactory is only loaded
    // once for all tests
    static {
        String pkg = ClassUtils.classPackageAsResourcePath(Util.class);
        String[] paths = {"classpath*:/" + pkg + "/applicationContext-*.xml",
        ctx = new ClassPathXmlApplicationContext(paths);
I got the block of code above from Matt Raible's AppFuse, a great project that uses Spring extensively.

The only change I made was to specify Util.class, which tells ClassUtils to return a string converted from the package name of the class. I don't really have any use for specifying the META-INF path yet, but I guess there is no harm leaving it in.

4. In my JUnit tests, I get the references to the objects I need by calling ctx.getBean().

5. In building the classes for running the unit tests, there are two approaches that I take - one is to build the project from within Eclipse, and another is to build the project from the command line using maven.

Somehow in Eclipse the applicationContext-*.xml files are copied to the target folder under the same location as that in the src folder, but in maven, I needed to specifically write a postGoal to perform the copying as follows:
  <goal name="copy.springconfiguration" 
  description="copy applicationContext-*.xml files to the appropriate package">
    <copy todir="${maven.build.dir}/classes/com/mydomain/util">
      <fileset dir="${basedir}/src/java/com/mydomain/util" 
includes="**/applicationContext-*.xml" />
  <postGoal name="java:compile">
    <attainGoal name="copy.springconfiguration" />

5. To enable the loading of the application context in the web app, I had to modify the web.xml file with the following additions:
  protected WebApplicationContext _springCtx = null;
  // Under setServlet, add
  _springCtx = WebApplicationContextUtils.getRequiredWebApplicationContext(getSerlvetContext);
7. In the action class that we need the Spring beans, we call the application context directory to return the object:
    private DateUtil dateUtil = null;
    dateUtil = (DateUtil) _springCtx.getBean("dateUtil");
where dateUtil is the id of the bean wired in the context file.

With these steps done, I can start to do IoC in seleted struts action files, and move the code gradually to Spring.
Posted on (2005-10-08 14:10:41.0) Permalink Comments [2]
20051006 Web 2.0 is going on
Thursday October 06, 2005
Category: General
The Web 2.0 conference is being held now in San Francisco. A little too far and too expensive for me to make the trip to be there physically, but I have been listening to podcasts from IT Conversations for last year's seminars. Really inspiring speeches (most of them), and makes me learn a lot more about all the new and exciting developments.

I must say that IT Conversations is one of the two podcast sites (the other being Adam Curry's DailySourceCode) that really got me hooked on the idea of utilising my "drive-time" with podcast listening. IT Conversations is funded by listener contributions, and I think I am really going to contribute to their TipJar. Kudos to IT Conversations executive producer Doug Kaye and his Team ITC volunteers for bringing such a valueable service to all!

Posted on (2005-10-06 04:03:38.0) Permalink Comments [0]
20051006 Google Maps - the first Web 2.0 platform
Thursday October 06, 2005
Category: General
Got this from the O'Reilly Radar article.

Has been looking cursory at Google Maps, but the issue that is stopping me from spending more time on it is that for where I living (Singapore), the map information is not as detailed as that available for the States.

shooperman showed my his installation of Google Earh today and I realised that the information is out there, but somehow it is not included in Google Maps. Anyone out there knows if it is possible to send a request to Google to include that information in Google Maps?
Posted on (2005-10-06 03:49:56.0) Permalink Comments [0]
20050928 Eclipse Tip: refresh plugins
Wednesday September 28, 2005
Category: General
I am sure this is a tip that has been going around for a while, but I think I should blog it down so that I can remember it in future when I encounter the same problem.

I recently upgraded to Eclipse 3.1 (only to find that Eclipse 3.2M2 is already released later, but that later), and wanted to use the Sysdeo Tomcat Plugin with e3.1. After downloading the new Sysdeo plugin file for e3.1, I place the unzipped folder in the plugin folder of e3.1, but the tomcat plugin was not detected.

Searched for a solution and I came upon this article that tells me the solution straight away: to start eclipse with java -jar startup.jar -clean. However, in my case, because I installed a copy of Oracle on my machine, and Oracle messed up my path by adding the path to a JDK1.3 installation at the front of the path, I have to start e3.1 with %JAVA_HOME%\bin\java -jar startup.jar -clean at the e3.1 folder, with %JAVA_HOME% pointing to the correct java SDK version I want.

The same command can be used to refresh eclipse whenever a new plugin is installed.
Posted on (2005-09-28 11:59:25.0) Permalink Comments [0]