20061010 Tuesday October 10, 2006

Adding Routes to the Framework

It wasn‘t hard adding routes to the framework, and of course I followed the Ruby on Rails lead. The goal is to avoid having to write a whole bunch of redundant code just because. I created “Route“ objects which took care of matching URLs and populating the Request object. Each Route object is initialized with a String representation of the URL fragment it responds to and an optional Map of default request attributes. For example, you can create a Route that responds to the root of the context “/“ and pass in default request attributes to specify the controller and action. Alternatively, you can have parts of the URL match a named attribute. For example the common url spec “/{controller}/{action}“ will match the browser provided URL “/user/login”, setting up the controller to be the UserController class (see my previous post) and the action to be the login() method. It‘s a much more flexible API, and it actually produces cleaner code.

The next part of the agenda was to set up the interceptor filters. The controller can be set up so that certain methods are called before or after other methods. While the traditional Ruby implementation allowed you to do all kinds of fun things with the parameters, Java just isn‘t as flexible. As a result I added several methods such as beforeFilter(filterName), beforeFilterOnlyFor(filterName, actionNames...), beforeFilterExceptFor(filterName, actionNames...) and the “afterFilter“ equivalents. The Filters also specify a specific interface with methods aptly named “before“ and “after“. The framework includes a default implementation that uses reflection to call the filter methods to satisfy the beforeFilter and afterFilter variants. It keeps the code very clean.

Next on my agenda is to add the beforeAndAfter filter methods that accept the traditional Filter interface. I like the flexibility of the Ruby implementation where there is no explicit interface. As long as an object has a “before()“ method and an “after()“ method, it can be used as a filter. I just may continue using that, and keep the Filter interface hidden. Then I want to set up the “urlFor” methods. That will handle the largest portion of useful features in the ActionPack.

(2006-10-10 09:14:04.0) Permalink Trackback Comments [0]

20061007 Saturday October 07, 2006

ActionPack for Java

I know I haven‘t posted anything for a long time, but when you are working in downtown DC and the security measures are so draconian you don‘t have any access to the internet it makes it more difficult to keep the blog updated. So, I‘m on another J2EE project and because of the same draconian security measures I have to reinvent the wheel for most things. I can‘t even install a proper JDK! I‘m just glad Eclipse has a compiler built in. Since I have to reinvent the wheel, I might as well do it the way I wanted.

I really like the way program logic is done in Ruby on Rails, and there is no technical reason why I can‘t do that for Java. In fact, the Java Servlet specification makes it fairly easy. There are two methods of dealing with requests. You can use a Filter and FilterChain or you can use a RequestDispatcher. They take care of different aspects. The Filter allows you to inject your business logic and manipulate the HttpServletRequest and HttpServletResponse objects rather well. The RequestDispatcher allows you to use the results of another servlet/JSP/file in your response without changing the URL.

I took the tack of building my Controller so that it operated in the Filter space. I invoked the controller using reflection, which worked like a charm. I even built in logic to return a 404 response if a requested action wasn‘t implemented. I also really liked how Rails controllers fell through to rhtml files if they didn‘t send a response right then. I used the RequestDispatcher to handle that logic. In my work downtown, I figured it was much easier to fall through to a JSP than anything more elaborate. Truth be told, my Controller infrastructure is rather agnostic about what really implements the response. It could be JSPs, a Cocoon instance, etc.

In fact it would make a wonderful addition to Cocoon to tame the wild beast. In essence, the Controller framework would live as a filter that works before Cocoon gets the request and Cocoon merely responds with a pipeline. No need for CForms or other heavy work.

The true power of the Rails integration has to do with integrating the ActiveRecord and ActionPack modules. I wish I had the time to really flesh that out downtown, but I simply got myself to a point where I can move forward and get the job done. I believe I‘ll start fleshing that out — still living at a Filter layer for most things. That gives us the best of all worlds. We can use whatever we want to handle the display aspects and be as simple as merely forwarding a file or as complex as a full XML pipeline.

All this coming to a repository near you….

(2006-10-07 10:20:40.0) Permalink Trackback Comments [3]

20060830 Wednesday August 30, 2006

Rapid Evaluation Madness

I‘ve wrapped up the last project using Ruby and ActiveRecord to extract big complex records about people and turn them into Wiki pages. Thanks to WikiMedia‘s XML import ../../../page/format__the_process_wasn__8216.css;t too difficult. While I learned more about Wiki markup than I really cared to, the real story with that was how to pull 12,688 records that each had no less than 16 associated sets of records and attachments using ActiveRecord and Microsoft SQL Server. Once I got the connection working (using ADO) I quickly ran into a problem of scale. The way ActiveRecord works is that it uses lazy initialization to build all the objects as you need them. With a set of 12,688 complex objects (and associated attachments) I would run out of memory after processing around 1,000 records — and I have 2GB on my machine.

The way around the high volume of data problem is actually pretty simple. ActiveRecord really wasn‘t designed for this type of problem, so I had to work around a couple of limitations. First, I can‘t have 12,688 complex objects in memory at the same time (some of those attachments were about 25MB and that data was loaded in as well). I can have 12,688 simple IDs in an array. So I used a simple query to get the IDs and then looped through looking up each person by itself. Worked like a charm.

Now I‘m on a rapid application evaluation/development project to incorporate a survey management tool into the government infrastructure. I have to slog through some documentation before I get down and dirty. In the process of setting up the server I ran into a particularly ugly snag. The SELinux extensions prevent the MySQL server RPMS from running. I‘ve seen a couple links, but I don‘t have the authority to download the patches I need. This is quite frustrating, but as the high level architecture documentation takes precidence I have to push it off a bit. My due date is tomorrow. Yes, government hoops are fun. You spend a lot of time creating paper for people to skim through and never look at again. Such is life as a contractor.

(2006-08-30 19:27:23.0) Permalink Trackback

20060722 Saturday July 22, 2006

Feedback, It's What's for Dinner

I may be an odd fellow. As I learn something new, I always want to know its origins. Where did it come from and how does it make our lives better. As part of this natural pull, I recently bought a 50 year old Twin Lens Reflex (TLR) camera. It was my cheapest option to enter the world of medium format photography, which promises better resolution and more incredible pictures. I haven‘t gotten the pictures back from the developer yet, so I can‘t say if the claims are true for this camera. However, I‘ve made a few observations.

As cameras have progressed, we have more and more feedback available to us. In the beginning, cameras were made without meters and you had to go by the “Sunny 16” rule. Without boring you with details, the sunny 16 rule is a guide to base your exposure on. Many photographers still swear by it and won‘t bother with modern gadgetry. But as cameras evolved, and the bikini was introduced (the biggest boon to camera sales in history), more and more cameras came equipped with exposure meters. Now you had visual feedback that would tell you what you should already know (the Sunny 16 rule). Only thing is that meters could get fooled. They judge how bright the light is by comparing the average intensity with a medium gray (18% to be exact). If the picture had too much white or black, the meter would give you erroneous feedback. Which is part of the reason some photographers carry around a callibrated gray card with them.

Things progressed and we also got feedback on whether we were in focus or not (within a certain margin of error of course, we are dealing with machines). Now in the digital age, we have the ultimate feedback: the captured picture. More and more gadgetry thrown at the problem of how do we make beautiful pictures? The gadgets can‘t make beautiful pictures, but they can offer possible reasons why it didn‘t come out like you may have wanted. And that‘s the topic for today: feedback.

I have to say, with a good feedback system in a framework or application it is much easier to diagnose what might be happening. I‘ve worked on quite a few systems with feedback that ranged from console messages to live monitoring of a running system. In each system the feedback you got could point you in the wrong direction. It could send you on a wild goose chase. That can be very frustrating, particularly when you have people wanting some sort of answer and you are no closer to having one than you were when you started. The bottom line is that your feedback system can be fooled. Even if your feedback system includes the final result (such as a digital picture), it can‘t tell you why the technically correct picture didn‘t impress you. It can‘t tell you if one composition is stronger than another. Some things are left to the person to answer.

So if your monitoring system can be fooled, how can you trust it? What do you do if you can‘t trust your instruments? Pilots have to answer that question. An airplane has very sophisticated instruments, and a skilled pilot can fly a plane successfully without looking through any of the windows. The pilot can also tell if an instrument is out of whack because the instruments have instruments and warning lights. In the event of a complete failure of the monitoring system, the pilot has to actually pilot the plane. They have to make decisions based on what they can see from the cockpit. With software, we might feel like we are flying the Wright brother‘s plane or you might feel like you are flying the most advanced stealth bomber. Either way, we need to know a certain amount of information at a glance.

The problem for us software engineers is deciding what instruments actually help. How can we tell what is going on in a running system? What is the most telling metric? Is it memory consumption? It might tell us if there is a memory leak somewhere. What about average time to process a request? Well that might only be useful in development. In my experience, the best metrics are the ones that have a definite cause and effect relationship. If the metric looks like X then I need to do Y to fix the system. If I have a compilation error, I need to fix the syntax of my program. If a unit test fails, I need to fix my program. If I get a NullPointerException, no wait you don‘t still get those do you? :)

Even when you have a set of clean and distinct cause and effect type of meters built into your system, things can still go haywire. It‘s at those times that you have to step back and pull from old-school sensibilities. I may not be old school in age, but I am in spirit. Of course you go down the list of things that could possibly be an issue. The amount of data, the amount of traffic, possible sequences of events. Eventually you have to go outside the realm of common issues. In a military institution, the computers in the control tower would all go down at the same time twice a day. It wasn‘t until an embedded programmer noticed the radar making a full sweep and frying all the electronics in its path at the exact same time the computers went down that they found out what the problem was. Yes, the answer can be that bazar.

I have four cameras now. A digital, two 35mm film cameras, and an old medium format camera. Of them all, I am most comfortable with my high end 35mm camera. It has just the sensors I need, and it is very accurate most of the time. Out of the several hundred pictures I have taken with it, only a small percentage have been not so great. I have a lot of trust in what it can do, and because of it I have less trust in my starter 35mm camera. In fact, I have less trust of my digital point and shoot camera. Most of its pictures are overexposed, and the focus isn‘t as sharp as I‘d like. I‘ve been told that my vintage medium format camera can give me better results, but until I have any kind of real feedback I won‘t know for sure. It does encourage you to think more about what you are doing before you do it.

At the end of the day, the one thing that helps to understand and diagnose issues more than anything else is to shorten the feedback time. Once you have a quick feedback, you can decide how helpful it is. The longer it takes to try/fail/diagnose the more frustrated you become. Sometimes going without a modern convenience can really improve the system. Sometimes you can retrofit the modern convenience into the older design. No matter how quick your feedback system is, if you can‘t trust what it tells you then it is useless. You have to know what can fool the system and how to compensate for it. With cameras it is relatively easy because everyone has the same basic answers regardless of whether they are using Canon, Nikon, Kodak, Rollei, or Hasselblad. The only differences have to do with what buttons you press, but not the type of adjustments you are making. Software is not nearly so simple. Many times the same symptom points to completely different issues based on whether you are using Microsoft, Oracle, or IBM. It‘s this disconnect that makes it difficult to build a stronger community of common issues and solutions. Remember that community is just a larger system that needs feedback too.

(2006-07-22 22:44:11.0) Permalink Trackback

20060626 Monday June 26, 2006

Using ActiveRecord for Export Purposes

In the main project we are working on, we have decided to cut our losses on a particular type of information we were gathering. Since a WIKI works just as well to house the information, and there really hasn‘t been any strong demand to mining the discrete information we have, we need to import our data into the WIKI. Luckily enough, Wikimedia (the software we are exporting to) has an export/import feature. That makes the process of generating uniform records relatively painless. The only catch is that we have to pull the data out of our database and into the WIKI format.

Since I love the ease of use of working with Rails, I decided to take the most useful parts and use them in my export wizard. The database is Microsloth‘s SQL Server, but thankfully there is an ADO bridge for Ruby. The ODBC bridge doesn‘t seem to work (memory issues), so do a search for using SQL Server with ActiveRecord. In fact the two parts that I am using from Rails is ActiveRecord and ERB integration. For ActiveRecord, I only need to configure the database connection and set up the model objects.

The database I am working with is an ugly hack that was once redesigned, and then because of schedule issues the redesign was scrapped. Naturally I could not use the golden path for ActiveRecord. However, that really isn‘t a huge problem because I can call set_table_name and set_primary_key to override those defaults. The trick is that I need to require all the ruby files in the models directory. Right now that is being done manually, but I aught to make it more robust.

Getting ERB working was probably the most fun. The source code below is the short form of what I needed to do:

require 'erb'
template_file = IO.read('views/page.rxml')
template = ERB.new(template_file)
Person.find(:all).each do |person|
@person = person
puts template.results

The results method re-evaluates the template into a rendered string each time. The real action performed here will be to output that string to a file. The cool bit is that the details of what the thing looks like can be manipulated easily in the views/page.rxml file. Because everything centers around that one central Person record, I don‘t have to add new variables for the embedded Ruby interpreter to use.

The approach I‘m taking is fun, so that helps make the task less of a pain. The really cool thing is that I whipped together the core framework in a couple of hours. The only real challenge is getting the models to relate to each other because of the poor design. Without intending to, ActiveRecord even made one of my tables polymorphic — something that was really the true intention of that table. This pretty much rocks.

(2006-06-26 23:28:47.0) Permalink Trackback

20060621 Wednesday June 21, 2006

Inherent Risk with Single Sign-On

At the time I am writing this, the Yahoo! single sign-on service is down. Completely down. No hope of logging in at all. In many sites, all that means is that you can‘t access your service but you can get other work done. In this case, I can‘t log in to Flickr, or access any of the other yahoo services. It‘s incredibly frustrating. The worst part is, I don‘t know who to notify or yell at. There is no indication that Yahoo! knows what is going on or is doing anything about it.

The risk is self-evident. Users are held hostage by the single sign-on service. Sure you have the convenience of signing in one time and then being able to go from place to place… Sort of. With Yahoo! and Flickr, you have to sign in again because the cookie has to be set at the domain level, and Flickr has its own domain. It‘s definitely less than ideal.

(2006-06-21 09:45:41.0) Permalink Trackback Comments [2]

20060616 Friday June 16, 2006

Zen: The Art of Disconnecting?

I spent more time contemplating why it is that I like the analog more than digital. I have a lot of time on my commute so thoughts just come bubbling up as I ride home. Then I thought of different arts where the principles of Zen are traditionally applied: flower arranging, gardening, sword polishing, martial arts, painting, haiku, and tea ceremonies. There is a common thread among the ideals in all of these disciplines. It‘s the practice of disconnecting your conscious mind and allowing your subconscious mind to surface. When you combine that with the modern theories on how our brains work, it makes an incredible amount of sense.

Our brains need rest, just like our bodies do. To much active thought can cause you burnout, stress, and/or panic. When you develop your own film, it does not require much in the way of active thought. When you have your workspace properly arranged, you only need to watch the clock to know when to agitate or change chemicals. After you‘ve done it a couple times, you have now carved out 20-30 minutes where your brain can dissengage from active thought and allow your passive thought to surface. The mind wanders all over the place. Sometimes it dwells on a moment of beauty that you saw and different aspects of that moment come to light. Other times it dwells on whatever challenge you were facing that day, and unique aspects come to light. In short, doing something physically repetitive that does not require too much active thought frees you to be creative.

At some point you may have a moment where your passive thought resonates with your active thought. A unique and creative approach to solve a hard problem. A way to bring more out of a picture. A small change to maximize beauty. Some people call it a “Zen“ moment, others call it a “Eureka!” moment, either way it would not come to the surface if your active mind is engaged all the time.

For thought professionals like us developers and designers, this underscores the need for us to walk away at the end of the day. Powering through a tough problem forces you down the wrong path. Allowing all the stuff you are dealing with to fall into the background helps your brain sort out all the information it has to deal with. You can see the path you couldn‘t see before. It‘s not mystical, it‘s the way we are wired.

(2006-06-16 10:19:56.0) Permalink Trackback Comments [2]

20060615 Thursday June 15, 2006

Analog vs. Digital

Last night I upgraded my zoom lens, bought film, and got harassed for not having a digital SLR. I know the ACE Photo people mean well, but I still love film. It's largely for the same reason that I love analog tape more than straight to digital. It's that factor of quality . I know the strengths and weaknesses of each medium. Yet, I like my results better with an analog front end than with a digital front end.

Digital is accurate, but it is not real . It is a quantized version of reality. Over time technology has given us more and more accurate representations of reality to the point where it is almost undiscernable. So why is it that I still prefer film and tape for the beginning of my work flow? One could argue in the case of film that if you apply the right ICC profile and such you can simulate the film. You could also argue that in the case of tape, it is just a combination of EQ and compression. Nevertheless, in both cases, you know exactly what you want and all you have to do is put in the right media.

There is another reason for preferring the analog processing for the beginning stages: softer saturation characteristics. When you over-saturate a film, you can rescue more than you can when you over-saturate your digital capture. Film gives a little more. The same goes for tape — even more so. When you over-saturate your tape to the point of distortion, the sound is more pleasing than digital.

Finally, as smooth and clear as DSLR cameras have become, there is an element of soul (deep feeling) that is lost. Digital is much more sterile feeling than the rawness of analog. Much of it is due to the absence of noise. Sometimes that is what you want, but many times you have a much better emotional response to the analog medium. Why do you think many audiophiles insist on vinyl records instead of compact discs? There is a better visceral feel to the realness of analog than the sterility of digital.

So, what does that mean for software? Isn‘t the medium we are working with purely digital? Well, yes and no. It all starts with people. People tell machines what to do and the machines respond. Some applications do a better job than others with this. The question is, how analog are you ? Have you fallen into the trap of thinking like a machine or do you vigilently protect your humanness? What you create depends on who you are internally. If you are sterile inside, then you will create sterile things — applications or “art”. If you are alive inside, your creations will be alive. Sometimes you have to leave the sanitarium :)

(2006-06-15 09:14:29.0) Permalink Trackback Comments [1]

20060609 Friday June 09, 2006

Getting Lay-People Developing--Maybe its the technology

In my humble experience, the more you try to hide from users the more frustration you cause. When you try to hide complexity and treat your application or framework as a black box, you are hiding several failure points. If the user experiences problems and everything looks kosher in their code according to examples and help they‘ve received, they will blame you for their problems. And rightly they should. They wouldn‘t be experiencing the frustration of problems they can‘t fix because the problem is hidden in a black box.

I had an interesting conversation this morning with a co-worker. He asked me which language/framework I would recommend to help him do a certain application. Considering his experience is mostly documentation writing with some DB experience I highly recommended Ruby on Rails. What I didn‘t expect was his reaction to it. Apparently, I undersold rails. He was so thrilled to get an example up and running in 15 minutes just following the tutorial that he told me I didn‘t do it justice with my description.

Bottom line is this: when your technology is simple enough for a lay-person to understand, they will use it. No black boxes needed. No weird hacks to cater to stupid people. Just plain well organized software. In fact, catering to stupid people is the worst thing you can do. It is why black boxes are so complex. If you assume your users will have a certain level of intelligence, they will actually rise to meet your expectations — within reason of course. If a trained developer with over 10 years of experience can‘t get how to go anywhere after 8 hours with your software then the problem is your software — not the developer.

The KISS methodology has served us well for many years, and more folks are just now getting the picture. Every advance in technology has to do with simplifying our lives. Keep it up. Push to find the next rails. Create something that people enjoy using.

(2006-06-09 10:41:56.0) Permalink Trackback

20060531 Wednesday May 31, 2006

Good and Efficient Software Development

At the end of the day, the only thing that matters in the software development world is that the project was finished accurately and on time (within an acceptable margin of error). All the different SD processes are peoples best guess at how to achieve that consistently. So far, the Agile principles (there is no formal process, but that is actually a strength) are the closest to ensuring accurate and efficient development. Communication is key, and any process that emphasizes communication is going to work better than one that emphasizes work artifacts.

The SD world consists of three aspects: design, development, and testing. But what about requirements? That‘s an aspect of design. To effectively manage an SD team, you have to manage how these relate to each other. Any process that emphasizes rigid design and requires a heavy process to make changes will fail . Design needs to be fluid and easily changed as the project progresses — right down to requirements. I know some people are scared of such a strong stance, but in my experience the customer appreciates the flexibility. My customers include both commercial companies and federal government, so don‘t throw the baby out with the bath water. The one aspect of a project that has the slowest rate of change are the project goals. All the requirements, design, and completed software are there to accomplish those goals. Your customer usually wants to know that if a proposed solution to meet a goal can‘t be done, you have an alternate solution to meet the goal. If you can show how it can even be better, they will be that much more appreciative.

I believe that you never finalize any part of design during the process of development. The impact of that is that you can never truly have a software design specification. I have yet to see a design spec worth the effort to create it. By the time the design is documented it has already changed — and that is true wether you have a heavy process for the changes or not. So what happens? The SDS becomes dead weight that no one updates and only exists to fill a checkbox on someone‘s process. Does that mean that UML or other diagramming is useless? Absolutely not. Many developers work well with pictures to help convey how things work together. However, those diagrams never stand on their own and always require someone to explain them. Even UML documents that are supposed to stand on their own. Those diagrams are then thrown away when everyone is on the same page. No need to keep them around and maintain them with versions.

Lastly, I believe that the detailed design is best documented with automated unit tests, or possibly the emerging spec based validation tools (an evolution of unit test systems). The bottom line is that a change is made to the automated validation system before it is made to the code. As useful as unit tests are, and as much as they have saved my butt to help me refactor or catch regressions, they are no replacement for a dedicated test team. You need someone who thinks differently than a developer to truly put your app through its paces. Your testers are an integral part of the software development team, and they should be used throughout the development cycle instead of only at the end. You can catch many design and implementation issues long before it starts impacting your schedule.

(2006-05-31 09:23:09.0) Permalink Trackback