Jeffrey Phillips Blog

Configuring Web Logic 10.3.5 and log4j for your Java Application

2012-01-25T15:37:00.000-08:00

I recently spent several days trying to track this down why Apache log4j for my Java application was working when deployed to Oracle WebLogic Server 10.3.5. I am posting my findings here in the hopes of saving others some time. I found several blogs that talked about this, but the end result was that log4j was still not working correctly. There was always one step missing. A co-worker and I finally figured it out for both Linux and Windows.

1. Copy wllog4j.jar from your WebLogic server/lib directory and place it in your domain_root/lib folder.
2. Copy your version of log4j.jar to your domain_root/lib folder.
3. Copy your log4j.xml to your domain_root folder.
4. Log in to the your WebLogic admin server console. Click on Servers -> Admin Server -> Logging. Click on advanced mode, and change the logging implementation from JDK to Log4J. Save your changes.

Most of the blogs had these items, but the one critical piece that was missing:
5. (Linux)Edit the setDomainEnv.sh file in the domain_root/bin directory. and find the following code:


if [ "${LOG4J_CONFIG_FILE}" != "" ] ; then
       JAVA_PROPERTIES="${JAVA_PROPERTIES} -Dlog4j.configuration=file:${LOG4J_CONFIG_FILE}"
       export JAVA_PROPERTIES
fi

Insert the following above those lines of code(replacing the path with the one for your system's domain_root directory:


LOG4J_CONFIG_FILE=”/u01/app/oracle/middleware/user_projects/domains/base_domain/log4j.xml”
Export LOG4J_CONFIG_FILE

5. (Windows)
Edit the setDomainEnv.cmd file in the domain_root/bin directory. and find the following code:


if NOT "%LOG4J_CONFIG_FILE%"=="" (                                                          
            set JAVA_PROPERTIES=%JAVA_PROPERTIES% -Dlog4j.configuration=file:%LOG4J_CONFIG_FILE%
)

Insert the following above those lines of code(replacing the path with the one for your system's domain_root directory:


set LOG4J_CONFIG_FILE=”C:\Oracle\Middleware\user_projects\domains\base_domain\log4j.xml”

6. Activate the changes and restart the admin server.

Here is a sample log4j.xml file. Obviously, you will need to update it for a valid location for your file appender, and replace yourcompany and yourproject with valid values.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
   <!--====================================================================-->
   <!--    Appenders     -->
   <!-- %d{dd-MMM-yy HH:mm:ss} %-5p [%c{1}] %m%n  -->
    <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="[%p] - %m (%F:%M:%L)%n"/>
            <!--param name="ConversionPattern" value="%d{ISO8601} %p [%c{1}] - %m (%F:%M:%L)%n"/-->
        </layout>
    </appender>

    <appender name="FILE" class="org.apache.log4j.RollingFileAppender">
        <param name="File" value="/temp/logs/yourproject.log"/>
        <param name="Append" value="true"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ISO8601} %t %-5p %c{2} - %m%n"/>
        </layout>
    </appender>

    <!--====================================================================-->
    <!--    Logging Levels     -->

    <category name="com.yourcompany.yourproject">
      <priority value="DEBUG"/>
    </category>

    <category name="org.apache">
        <priority value="WARN"/>
    </category>

    <category name="org.springframework">
        <priority value="WARN"/>
    </category>

    <!--===================================================================-->
    <!--    Appender Assignments     -->
    <root>
        <priority value="ERROR"/>
        <appender-ref ref="FILE"/>
    </root>
</log4j:configuration>

Again, I hope that this will save someone the pain I went through to track this down.

Exploring Spring Integration & Spring Web Service, Part 1

2011-11-22T14:31:00.000-08:00

I have been learning Spring Integration and Spring Web Services on the project I am on. The more I learn the more I like the framework, but I have to say it has not been easy to make progress. The number of resources available is lacking. I thought I would spend the next several blogs covering some of the information I have been learning.

I needed to do was expose four SOAP Web Services based on a set of WSDL and XSD files provided by the vendor. First thing that had to happen was setting up the web.xml file. From reading the documentation on the main Spring site, I determined to use MessageDispatcherServlet.

web.xml


<web-app>
   <display-name>My Application Name</display-name>
   <servlet>
       <servlet-name>si-ws-gateway</servlet-name>
       <servlet-class>org.springframework.ws.transport.http.MessageDispatcherServlet</servlet-class>
       <init-param>
           <param-name>contextConfigLocation</param-name>
           <param-value>classpath:si-ws-gateway-config.xml</param-value>
       </init-param>
       <load-on-startup>1</load-on-startup>
   </servlet>
   <servlet-mapping>
       <servlet-name>si-ws-gateway</servlet-name>
       <url-pattern>/*</url-pattern>    
   </servlet-mapping>
</web-app>

Next, let's take a look at the spring application context file.
si-ws-gateway-config.xml


    <bean class="org.springframework.ws.server.endpoint.mapping.UriEndpointMapping">
        <property name="mappings">
            <props>
                <prop key="${data.subscriber.url}">dataGateway</prop>
                <prop key="${request.status.url}">statusGateway</prop>
        </property>
    </bean>

    <si:channel id="dataArrivedChannel" />
    <si-ws:inbound-gateway id="dataGateway" request-channel="dataArrivedChannel" />
    <stream:stdout-channel-adapter id="stdoutAdaptorWithDefaultCharsetDataArrived" channel="dataArrivedChannel"/>

    <si:channel id="statusChangedChannel" />
    <si-ws:inbound-gateway id="statusChangedGateway"
                           request-channel="statusChangedChannel"
                           marshaller="marshaller"
                           unmarshaller="marshaller"/>
    <si:service-activator input-channel="statusChangedChannel"
                          ref="statusServiceEndpoint"
                          method="processStatusMessage"/>   

    <bean id="marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>com.ourvendor.StatusUpdate</value>
            </list>
        </property>
    </bean>

    <bean id="StatusDao" class="com.company.StatusDao"/>

    <bean id="statusServiceEndpoint"
          class="com.company.StatusServiceEndpoint">
        <constructor-arg ref="statusManager"/>
    </bean>

    <bean id="statusManager"
          class="com.company.StatusManagerImpl">
        <constructor-arg ref="statusDao"/>
    </bean>

Well that seems like a lot of spring configuration, but it eliminates a bunch of Java code. Let's break down this configuration a bit. First, the UriEndpointMapping bean. You might notice that there are some values provided from a properties file. I will be covering properties in more detail in a future blog. For now, just know spring replaces the ${data.subscriber.url} and ${request.status.url} with a valid URL value from a properties file. This defines which URLs will be routed to the specified spring integration web services inbound-gateway. The gateway will place the SOAP message onto the specified channel as a DOMSource object.

As seen above the dataGateway does not specify a marshaller and unmarshaller which is why the DOMSource object is placed on the channel. The statusChangedGateway does specify the marshaller and unmarshaller, so when it places the object on the channel, it is not a DOMSource object, but a StatusUpdate object. As you can see it is very easy to leave the input in it's native format or convert it to a Java Object. In one case the object was only going to be routed(eventually) to a JMS queue, and did not need to be unmarshalled, in the other case it required unmarshalling so that a service activator endpoint could save the object to the database. As we will see in a future blog, it would have been nice if there was a way to specify the output to the channel as a string, instead of DOMSource. This would be a great enhancement to spring integration.

The service activator is an endpoint that allows a specific method to be called, and passes the object from the incoming channel to the method. It can also place the object, either unmodified or modified, onto an outgoing channel to be further processed by the system. The rest of the configuration is very straightforward spring beans with dependency injection. This works very well if each of the web services have a unique URL, but what happens if all calls come into a single URL. That will be the topic of the next blog in this series.

Accessing External Properties Files in WebLogic

2011-11-08T13:43:00.000-08:00

It is not only common to do, but is also a very good practice to put values that may need to change into a properties file. This creates a central location to modify values. Especially for the configuration a system based on the installation location, etc. Having the properties file inside the .war is very inconvenient when you want to change it as it causes you to recompile your code, even if it hasn't changed. I was on a project that was able to pull the property file out of the .war and put it in the JBoss AS conf directory. The application could still access it. Unfortunately the project I am on now uses WebLogic. After lots of google searching and asking a few co-workers if they new how to do it, I did find the answer.

Simply place the properties file in $WEBLOGICHOME/user_projects/domains/mydomain where mydomain is the name of he domain your app uses.

Works great and now I can change properties all day long without recompiling.

Remote Desktop & Change Password

2011-06-28T12:17:00.000-07:00

I ran into a problem where I needed to change the password on an account that was on a server half way around the world. All I could do was Remote Desktop in to that box, and when I did Ctrl + Alt + Delete, it was changing the password on the box I was physically at, not the one in the RD session. After some searching I found this solution: Ctrl + Alt + End.

Hope this helps someone.

Java JDBC Performance vs. iBatis

2011-03-28T11:00:00.000-07:00

While working on a recent project, I worked with a database that contained 23 billion rows in one table, and several hundred million in another table. Performance retrieving information was critical for success. I have used Spring and iBatis for several years now and only had a couple of weeks to develop a prototype, so implemented the database access layer using iBatis and spring. It is quick to implement and easy to maintain. We had several use cases we wanted to test. Some use cases returned as few as one row, some returned as many as 20 million. Unfortunately, it was taking too much time to retrieve the information for most of the use cases. One downside of using iBatis, was that I couldn't determine if the time was spent in the database query, returning the results to Java, or in creating the java objects themselves. I decided to modify the code to use straight JDBC calls so that way I could time the retrieval separately from converting to Java Objects.

I did not expect to see much performance improvement converting to straight JDBC. However, just converting to straight JDBC was a substantial performance improvement over using iBatis(30-40% faster). Unfortunately this still wasn't enough. This was a web application being deployed to WebLogic. After some research a co-worker found that the default number of rows retrieved by the WebLogic data source was 10. Queries whose results were less than 10, performed very well, but most queries returned thousands or millions of rows, so performance was very poor. We initial bumped that value to 1000, and saw another 40+% performance improvement. This in turn prompted some investigation of JDBC, and I found that the default value for JDBC against an oracle database was 10. Inside the code I increased the fetch rows value to 1000, and saw another 30% performance improvement. I concluded that more was better on the fetch rows setting, so I increased it to 50,000 and found that I now had a new problem: running out of memory. After some additional modifications to the memory available to the web application, and reducing the fetch count to 5000, I was able to get a reliable result. Essentially, a higher fetch count, meant less database connections to retrieve the data.

Using iBatis is a great way to get a lot done in a short amount of time, and for smaller databases I still prefer to use it. In addition to being easy to implement, it is also easy to maintain. However, if you have a large database, then using Straight JDBC might be a good solution. Don't forget to take fetch size into account both in the application server and in the Java JDBC code when dealing with large databases. The larger the fetch size, the fewer database connections that will be used, but the more memory that will be required. I did not investigate setting the fetch size in iBatis, so do not know if that is possible.

In the Java JDBC code setting the fetch size is done as follows:


            String query = "SELECT * FROM my_table";
            Connection conn = _dataSource.getConnection();            
            PreparedStatement st = conn.prepareStatement(query);
            st.setFetchSize(5000);
            ResultSet rs = st.executeQuery();

Some other Java improvements that helped with performance, was reducing the number of java objects being created and destroyed. This includes making any strings that are reused static member variables so they are only created once and reducing the number of java objects that are being created. If you are outputting to a file or or a message, perform those operations inline to reduce creating intermediate objects.

Jersey and Spring Injection

2010-07-28T11:00:00.000-07:00

On the project I am on we use iBatis, Spring, and Jersey. We have been using spring injection for our DAOs since the beginning, but our version of Jersey at the time did not support spring injection. To get around this we created factories allowing us to inject beans into the factories. Our REST classes then used the factory to get the beans they were interested in. As the project grew, so did the factory. When we upgraded to Jersey 1.0.3, we were able to take advantage of a Jersey-Spring jar, that would allow us to spring inject beans directly into our REST classes.

In our Maven pom.xml file we added the following dependency.


        <dependency>
            <groupId>com.sun.jersey.contribs</groupId>
            <artifactId>jersey-spring</artifactId>
            <version>1.0.3</version>
        </dependency>

Then in the constructor of the REST Classes we were able to do the following: (NOTE: as of Jersey 1.4, @Inject becomes @InjectParam)


public class CompanyWS {
    private ICompanyManager _companyManager;
    private ICompanyTypeManager _companyTypeManager;
    public CompanyWS (@InjectParam("companyManager") ICompanyManager companyManager,
                      @InjectParam("companyTypeManager") ICompanyTypeManager companyTypeManager) {
        _companyManager= companyManager;
        _companyTypeManager= companyTypeManager;
    }
...

The web.xml file also needs to be modified to call out the SpringServlet, instead of the ServletContainer:


    <servlet>
        <servlet-name>REST Application</servlet-name>
        <servlet-class>com.sun.jersey.spi.spring.container.servlet.SpringServlet</servlet-class>
        <init-param>
            <param-name>javax.ws.rs.Application</param-name>
            <param-value>com.accenture.netcds.api.rest.RegisterResources</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

One downside of the 1.0.3 Spring-Jersey library is that the automatic finding of the REST classes does not work in websphere 6.1. This has been reported as a bug to Jersey and is supposed to be fixed in the more recent versions. As a result the REST classes need to be registered in an Application class.


public class RegisterResources extends Application {    
    public Set<Class<?>> getClasses() {
        Set<Class<?>> s = new HashSet<Class<?>>();        
        s.add(CompanyWS.class);        
        return s;
    }
}

Spring successfully injected the beans into the REST classes, allowing us to delete the factories we had been using that had grown quite large. Another simplification to make it easier to maintain the software over time.

Nested iBatis Result Mappings

2010-07-28T10:00:00.000-07:00

Although, this is not an earth shattering iBatis discovery, it is a time saver, both in development and maintenance so thought I would share it.

Let's assume we have a class called CompanyType that contains type_id and type name and another class called Company that among other properties has a CompanyType object companyType included.

In the iBatis xml file the result map for CompanyType would look something like


<sqlMap namespace="CompanyType">
    <resultMap id="companyTypeResultMap" class="com.sample.CompanyType">
        <result column="company_type_id" property="typeId"/>
        <result column="company_type_nm" property="typeName"/>
    </resultMap>
...

In the ibatis xml file for Company the result map would look like:


<sqlMap namespace="Company">
    <resultMap id="companyResultMap" class="com.sample.Company">
        <!-- other company properties here -->
        <result column="company_type_id" property="companyType.typeId"/>
        <result column="company_type_nm" property="companyType.typeName"/>
    </resultMap>

The problem with this is that if something changes in company type, you have to know to change it in two locations. Makes it harder to maintain. Instead though you could do the following:


<sqlMap namespace="Company">
    <resultMap id="companyResultMap" class="com.sample.Company">
        <!-- other company properties here -->
        <result property="companyType" resultMap="CompanyType.companyTypeResultMap"/>
    </resultMap>

As you can see, now if the result map is updated in CompanyType, the changes are automatically reflected in the Company ibatis file. Hope you find this helpful. I know I did.

ExtJS store.find vs store.findExact

2010-03-29T08:40:00.000-07:00

I recently ran into a problem with an ExtJS combo box, specifically using the find method on the store. In the application, between uses, the last selected value was remembered so it could be selected automatically the next time the user logged in to the system. The problem was that the previously selected value was no longer a valid value. In this case, the behavior was supposed to default back to the first value in the list. However, it did not do that.

After some investigation, I found that store.find was doing a partial match, and in this case there were two names very close to each other. Let's say the last value selected was "War", and there was another value called "War Room". If the list was unsorted, it is possible that War Room would be earlier in the list than War, and it would get selected over War. Another scenario, is that between uses, the value of "War" is removed from the system. The next time the user logs in it would select War Room automatically, instead of the first value in the list. While the second is not as bad of situation as the first, if that behavior is unintended, it is unwanted.

During the investigation, I found that there was now a store.findExact method for store. I replaced the store.find method with store.findExact and that cleared the problem right up. Just wanted to make sure that others were aware of this. I don't remember ExtJS 2.x having this problem.

Lost Milliseconds using Date and Timestamp

2010-03-15T09:55:00.000-07:00

While trouble shooting some date discrepancies in our application, we discovered we were losing milliseconds, when converting from java.sql.Timestamp, to java.util.Date. The following test isolated the problem:


        DateFormat dfm = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");

        try {
            Date datea = dfm.parse("2009-05-13 11:11:03.113");
            Date dateb = dfm.parse("2009-05-13 11:11:03.257 ");

            assertTrue(dateb.after(datea));
        } catch (Exception e) {

        }

        Timestamp ts1 = new Timestamp(2010, 3, 15, 9, 9, 9, 111000000);
        Timestamp ts2 = new Timestamp(2010, 3, 15, 9, 9, 9, 222000000);
        assertTrue(ts2.after(ts1));

        Date datea = (Date)ts1;
        Date dateb = (Date)ts2;
       // NOTE this asserts false because it truncates the milliseconds in the cast.
        assertFalse(dateb.after(datea));

I am surprised that Java did not handle this. They could have provided a constructor that would take a java.sql.Timestamp as an argument, so they could preserve the milliseconds. Another solution was they could have kept both the milliseconds field and the nanoseconds field updated, so that the cast could have worked. Either way, hopefully, this saves someone else the time of tracking this down.

iBatis "Cache Miss", Part 3 Confguring the CacheModel

2010-03-01T12:30:00.000-08:00

In this final blog of this series we want to analyze the iBatis CacheModel, and how to configure it correctly. The first blog of the series discussed how to use JBoss logging to verify the problem and the previous blog demonstrated how to write a test to automatically test caching.

Let's start with simple example from the other blogs. The location_type table. It has no foreign keys to other tables. We only have to worry about changes to this table. Let's examine the location_type iBatis CacheModel.


    <cacheModel id="locationType_cache" 
                type="LRU" 
                readOnly="true" 
                serialize="false">
        <flushInterval hours="24"/>
        <flushOnExecute statement="LocationType.insertLocationType"/>
        <flushOnExecute statement="LocationType.updateLocationType"/>
        <flushOnExecute statement="LocationType.removeLocationType"/>
        <property name="size" value="10"/>
    </cacheModel>

There are four attributes defined for the CacheModel. The first is the id, this is the name you will reference on the select definitions in the iBatis XML file. The next is, type, which is the type of caching used. There two choices: LRU - Least Recently Used and FIFO - First-In-First-Out. We chose LRU so the most frequently used objects would be kept in the cache. The next two parameters are the two that determine how the cache model works(or doesn't if misconfigured).

I ran across a blog, that helped me understand what I was doing wrong. As Clinton explains how the readOnly and serialize attributes play together. I will replicate the meat of his analysis here, but have included the link to his post.

Setting readOnly to false, which means it can be modified, and serialize to false the data is not able to be serialized forces iBatis to limit the caching to the current session request. This essentially means that a second request from the client, will not use the cache. This was the situation we found ourselves in.

There are two possible configurations that will allow caching to work. You will have to decide which is appropriate for your project. The first is readOnly=true, serialize=false. This will cache the objects for in one cache accessed by all users. This assumes that the objects are readOnly and will not change, or that if they change you will have the appropriate flushOnExecute statements defined.

The second configuration you can use is readOnly=false, serialize=true. This will allow each user to have their own cache. This allows the tables to be updated with out negatively affecting the other users. The downside of this approach is memory usage. With each individual having their own cache, you can run yourself out of memory.

On our project we went with the single cache shared among all users. This placed the burden on us to manage the cache model. How do you do that? There are some other properties that can be set for the CacheModel. One of them is size of the cache for a given table. It will keep the most frequently used objects in the cache up to the quantity specified. Other parameters, include flushInterval. This allows you to say if this hasn't been flushed in N hours, flush it anyway. The finest control is telling the cache model what statements to flush on, using the flushOnExecute properties.

How do you know what statements need to cause the cache to flush? If it caused the database to change, you have to flush the cache. If you add a new record, modify an existing record, or delete a record you need to flush the cache. By examining the CacheModel definition above, you will see that it does just that.

That was the simple case. What if your table joins to other tables in it's select statements? Then cache management becomes more complex. You need to flush not only when your table contents are modified, but also when the contents of any table you are joined to changes. Consider the following CacheModel:


    <cacheModel id="locations_cache" 
                type="LRU" 
                readOnly="true" 
                serialize="false">
        <flushInterval hours="24"/>
        <flushOnExecute statement="Location.insertLocation"/>
        <flushOnExecute statement="Location.updateLocation"/>
        <flushOnExecute statement="Location.deleteLocation"/>
        <flushOnExecute statement="LocationType.insertLocationType"/>
        <flushOnExecute statement="LocationType.updateLocationType"/>
        <flushOnExecute statement="LocationType.removeLocationType"/>
        <property name="size" value="250"/>
    </cacheModel>

Location contains a reference to what its location type is, and subsequently all the selects performed on location join to location type to get the type information. We ran into a problem right after we enabled caching that when locations were cached, and the location_type had the name updated, it would not be reflected in the cache for 24hrs. After investigating we realized that the location cache was not being cleared out when location types were modified.

This forced us to start using iBatis namespace so we could reference one iBatis XML statements within another. In the CacheModel above you can see that the cache will be flushed anytime the location or location type table is modified.

Our project uses Spring 2.5.6 and iBatis 2.3.4. As a result of this finding, we also found a problem with iBatis 2.3.4. It does not support transaction caching. What this means is that, even though spring rolls back the transactions after each test method, if the cache was updated it is not flushed. We discovered this when running the automated test suite and they were failing.

To see this problem you can write a test class that has two tests. In the first test save a new record, and validate that it was saved. At the end of the test the database is rolled back. However the cache will still contain the record. in the second test method, do a get all and record the number of records returned. Let's say it was 2. Save a new record, which forces the cache to flush and get all the records again. If all was working as expected, the count would now be 3. However, the cache still had the record from the first test in it, when the save is done and the cache is flushed, the get will again return 2.

That's with tests. Can this happen in a live system? The answer is yes. If the saves are complex in nature(ie. more than one dao call required to save the information) and their are select's performed at different points during the save process that update the cache, if an error occurs before the save has completely succeeded and the transaction is rolled back. The database will be cleaned up, but the cache will still contain the records it attempted to save.

iBatis 3 is supposed to solve that problem, however, there is no Spring support for iBatis 3 yet. They have it planned for Spring version 3.1. For now, you must be very careful using cache on complex save operations.

iBatis "Cache Miss", Part 2 Creating automated tests

2010-03-01T12:25:00.000-08:00

In part 1, of this series we discussed how to set up the JBoss logging so that you can verify whether or not your iBatis caching is working. Once you get caching working, how do you prevent it from being broken by future modifications. It is important to create some automated integration tests, that will fail if the caching is broken.

You may ask does this make sense, after all iBatis is a third party Jar. Why would I want to test their code? If you can't trust third party jar files, where does the testing end. This is a valid argument. After all you would never write a test to verify that Java set/get methods work as expected. However, in the case of iBatis caching you are not so much testing third party software, although it will certainly do that, as you are testing that you configured the third party software correctly. As an added benefit, if you updated iBatis and they had broken the cacheModel, you would know immediately.

Spring provides a autowire capability for injecting beans into your test classes for integration tests. So we wanted to simulate the client behavior in an integration test. We injected the dao bean for a simple table in the database. We wrote two tests.

In the first test, we performed a get immediately that should have set up the cache. Next, making use of the Spring SimpleJdbcInsert class we bypassed our iBatis bean, to do an insert into the table. We did this so that iBatis would not flush it's cache on insert. Now if the cacheModel is set up correctly, when you go get the list of records again, it will retrieve it from the cache and it will not include this new record. Next, using the iBatis dao bean, we saved another record to this table. Now when the get all command is executed it should grow by 2 records(the one inserted through spring, and the one inserted through iBatis). Here is the test method.


    public void testDatabaseCaching() {
        List<LocationType> cachedTypes = _locationTypeDao.getAll();
        int numTypes = cachedTypes.size();

        SimpleJdbcInsert lJdbcInsert = new simpleJdbcInsert(_dataSource)
         .withTableName("location_type")
         .usingGeneratedKeyColumns("location_type_id");
        Map<String, Object> lParameters = new HashMap<String, Object>();
        lParameters.put("location_type_nm", "cacheTest");
        lParameters.put("location_type_desc", "verify ibatis caching working");
        lParameters.put("message_resource_key", "location.type.cacheTest");
        lParameters.put("active_flag", 1);
        lParameters.put("create_user", "ibatisCacheVerification@daoTest.com");
        lParameters.put("create_ts", Calendar.getInstance().getTime());

        // verify that the straight JDBC insert works
        Number lNewId = lJdbcInsert.executeAndReturnKey(lParameters);
        assertNotNull("autogen key shouldn't be null", lNewId);

        // ibatis cache should be unaware that we added a new 
        // location type since we did it with straight JDBC
        cachedTypes = _locationTypeDao.getAll();
        assertEquals(numTypes, cachedTypes.size());

        LocationType newType = new LocationType("cache2", "verify caching working part 2");
        newType.setCreateUserName("ibatisCacheVerification@daoTest.com");
        _locationTypeDao.save(newType);

        // insert causes flush, so cache should now have both new types
        cachedTypes = _locationTypeDao.getAll();
        assertEquals(numTypes + 2, cachedTypes.size());
    }

Running this first test prior to fixing the cache model resulted in assert failures, which validated that our caching was broken.

The second test was to validate a more complex caching problem. In this example, you have a location table with a foreign key to the location_type table. Performing a select on location will result in returning the location information as well as information about the location type. Suppose that the location select is cached, and then a modification occurs where some of the location type information is updated. If the cacheModel is configured incorrectly, it will not flush the cache and the location select cache will still contain the old values from the location_type table. Here is what that test looked like.


    public void testIbatisCacheWithJoins() {
        // get location types
        List<LocationType> cachedTypes = _locationTypeDao.getAll();

        // Add new location type
        LocationType locType = null;
        for(LocationType type : cachedTypes) {
            if (type.getId() == 1) {
                locType = type;
                break;
            }
        }


        // get locations to set the cache initially
        _locationsDao.getLocationsByTypeId(null, locType.getId(), null);


        locType.setUpdateUserName("ibatisCacheVerification@daoTest.com");
        locType.setName("New Type Name");
        _locationTypeDao.save(locType);
        List<Location>cachedLocations =  
           _locationsDao.getLocationsByTypeId(null, locType.getId(), null);

        assertEquals("New Type Name", cachedLocations.get(0).getLocType());
      }

Running this test case with the cacheModel set up incorrectly resulted in an assert failure as well. Now that we have the two main tests in place we needed to fix the CacheModel definition, which is discussed in the final blog, iBatis "Cache Miss", Part 3.

iBatis "Cache Miss", Part 1 - Validating the problem

2010-03-01T12:20:00.000-08:00

There is a lot of information to cover on this topic, so this will be a series of three blogs. First a little background. The project I am on, noticed some performance issues as we had more and more users. As a result, we decided to cache the heavily used, infrequently modified iBatis selects. At the time a lot of investigation was done to verify that the cache models were set up correctly and working.

Unfortunately, all the testing was done manually. So as fate would have it, subsequent changes ended up breaking the caching. However, due to upgrades and other optimizations in the hardware and software it was not immediately obvious that this had occurred. As a result, the project progressed for nearly a year with caching broken.

In this first blog I want to discuss how we were able to verify that caching was broken through JBoss logging. The second blog will discuss how to write an integration test that can be automated using maven, to prevent the caching from being broken in the future. The final blog, will cover a discussion of the iBatis cacheModel and how to configure it.

First, you will need to modify the jboss-log4j.xml file in the $JBOSS_HOME/server/default/conf directory. You will need to add a new appender definition as well as set up the categories you want to be output to that file. This information was found in this post. Here are the exerts from that file.


   <appender name="SQL_FILE" 
      class="org.jboss.logging.appender.DailyRollingFileAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="File" value="${jboss.server.log.dir}/sql.log"/>
      <param name="Append" value="false"/>
      <param name="DatePattern" value="'.'yyyy-MM-dd"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
      </layout>
   </appender>

   <category name="java.sql" additivity="false">
     <priority value="DEBUG" />
     <appender-ref ref="SQL_FILE"/>
   </category>

   <category name="com.ibatis" additivity="false">
     <priority value="DEBUG" />
     <appender-ref ref="SQL_FILE"/>
   </category>

When verifying this we started with our simplest example. A single table that did not join to other tables. From the client we brought up our admin interface for that table. This performed the initial select of everything which set up the cache. Refreshing, without having performed an insert, update, or delete should have returned the same results from the cacheModel. Much to our surprise, what we saw was "Cache Miss".


2010-03-01 13:43:56,515 DEBUG [com.ibatis.sqlmap.engine.cache.CacheModel] Cache
                   'LocationType.locationType_cache': cache miss
2010-03-01 13:59:41,921 DEBUG [java.sql.Connection] {conn-102346} 
                   Preparing Statement:          SELECT ...
2010-03-01 13:59:41,921 DEBUG [java.sql.PreparedStatement] {pstm-102347} 
                   Executing Statement:          SELECT ...
2010-03-01 13:59:41,921 DEBUG [java.sql.PreparedStatement] {pstm-102347} 
                   Parameters: []

Now that we have been able to manually verify that the cacheModel is broken, we need to write some automated tests so this doesn't happen again. That is the subject of of the next blog, iBatis "Cache Miss", Part 2.

After fixing the cacheModel the result was quite different.


2010-03-01 14:01:28,109 DEBUG [java.sql.Connection] {conn-102351} Connection
2010-03-01 14:01:28,109 DEBUG [com.ibatis.sqlmap.engine.cache.CacheModel] Cache 
    'LocationType.locationType_cache': retrieved object 'LocationType name: ...'

As you can see from the above log entries, it retrieved the results from the cache. Now when you do an insert, this is the output you will see:


2010-03-01 14:06:20,421 DEBUG [java.sql.Connection] {conn-102352} Connection
2010-03-01 14:06:20,421 DEBUG [java.sql.Connection] {conn-102352} 
       Preparing Statement:          INSERT ...
2010-03-01 14:06:20,421 DEBUG [java.sql.PreparedStatement] {pstm-102353} 
       Executing Statement:          INSERT ...
2010-03-01 14:06:20,421 DEBUG [java.sql.PreparedStatement] {pstm-102353} 
       Parameters: [Weather, null, location.type.weather, 1, jp1@test.com]
2010-03-01 14:06:20,421 DEBUG [java.sql.PreparedStatement] {pstm-102353} 
       Types: [java.lang.String, null, java.lang.String, java.lang.Integer, java.lang.String]
2010-03-01 14:06:20,703 DEBUG [com.ibatis.sqlmap.engine.cache.CacheModel] 
       Cache 'Location.new_locations_cache': flushed
2010-03-01 14:06:20,703 DEBUG [com.ibatis.sqlmap.engine.cache.CacheModel] 
       Cache 'LocationType.locationType_cache': flushed

Reviewing these log statements you can see that both the Location and LocationType cache was flushed, when the insert was performed. The next select that is done will hit the database again and cache the values returned.

Bulk saving and updating using iBatis

2010-02-28T23:08:00.000-08:00

Recently I had the need to save records to an association table. All the object values were the same except one. I was looping through the array list saving them one at a time, but kept getting unable to get database connection errors. I lamented that I couldn't do a bulk save. A co-worker thought that iBatis supported what I wanted to do, so after some investigation we were able to get both a bulk save and a bulk update to work.

For our example let us assume we have a table that contains a list of companies and another table that contains a list of Locations. From the client when you edit a company it allows you to multi-select a list of locations to associate with that company. On the back end you could loop through the locations and store each separately, but iBatis allows you to save them all at once. Let's examine how to do that.

Here are the Company and Location classes, minus getters/setters and imports;


package com.projectA;
public class Company {
    private int companyId;
    private String name;
    private List locations;
}

package com.projectA;
public class Location {
    private int locationId;
    private String address;
    private String city;
    private String state;
    private String zipCode;
    private String phoneNumber;
}

The database contains a table containing all the companies, and a table containing all the locations. There is also a third table that contains the company - location associations. To do this we need to take advantage of the iBatis dynamic fields. Here is the iBaits XML file that contains the bulk save and update methods for the association table:


<sqlMap namespace="Company">

    <typeAlias alias="company" type="com.projectA.Company"/>
    <typeAlias alias="loation" type="com.projectA.Location"/>

    <resultMap id="companyResultMap" class="company">
        <result column="company_id" property="companyId" />
        <result column="company_name" property="name" />
        <result column="{companyId=company_id}" property="locations"
                   select="Company.getActiveLocations"/>
    </resultMap>

    <resultMap id="locationResultMap" class="location">
        <result column="location_id" property="locationId" />
        <result column="location_address" property="address" />
        <result column="location_city" property="city" />
        <result column="location_state" property="state" />
        <result column="location_zip_code" property="zipCode" />
        <result column="location_phone_number" property="phoneNumber" />
    </resultMap>

    <sql id="select_company_fragment">
        SELECT company_id,
               company_name,
        FROM company
    </sql>

    <sql id="select_location_fragment">
        SELECT cl.location_id,
               loc.location_address,
               loc.location_city,
               loc.location_state,
               loc.location_zip_code,
               loc.location_phone_number
        FROM company_location cl, location loc
        WHERE cl.company_id=#companyId#
          AND cl.location_id = loc.location_id
    </sql>

    <select id="getAllCompanies" 
               resultMap="companyResultMap">
        <include refid="select_company_fragment"/>
        ORDER BY company_name
    </select>

    <select id="getActiveLocations" 
               resultMap="int">
        <include refid="select_location_fragment"/>
          AND active_flag = 1
        ORDER BY location_state, location_city
    </select>

    <insert id="bulkInsertCompanyLocation" parameterClass="map">
        INSERT INTO company_location
            (company_id,
            location_id,
            active_flag)
        SELECT
            #companyId#,
            #activeFlag#,
            loc.location_id,
        FROM location loc LEFT JOIN company_location cl ON cl.company_id=#companyId# 
                                                AND cl.location_id = loc.location_id
        WHERE  cl.company_id IS NULL
        <dynamic>
            <isNotNull property="locations">
                <iterate property="locations" 
                         prepend="AND " open=" loc.location_id IN ("
                         close=")" conjunction=",">
                    #locations[]#
                </iterate>
            </isNotNull>
        </dynamic>
    </insert>

    <update id="bulkUpdateCompanyLocation" parameterClass="map">
        UPDATE company_location
        SET
           active_flag = #activeFlag#
        WHERE company_id=#companyId#
        <dynamic>
            <isNotNull property="locations">
                <iterate property="locations" prepend="AND " open=" location_id IN ("
                         close=")" conjunction=",">
                    #locations[]#
                </iterate>
            </isNotNull>
        </dynamic>        
    </update>

</sqlMap>

The map that is passed into both the insert and update methods, contains the company_id, an active flag, and an array list of locationId's. These methods save or update multiple records in one shot. That's all there is to it. By changing to the bulk insert and update methods, the out of database connections errors went away.

The resulting SQL that iBatis generates looks like this:


   INSERT INTO company_location
       (company_id,
        location_id,
        active_flag)
   SELECT
       #companyId#,
       #activeFlag#,
       loc.location_id,
   FROM location loc LEFT JOIN company_location cl ON cl.company_id=#companyId# 
                           AND cl.location_id = loc.location_id
   WHERE  cl.company_id IS NULL
     AND loc.location_id IN ("1","3","10")

   UPDATE company_location
   SET
      active_flag = #activeFlag#
   WHERE company_id=#companyId#
     AND location_id IN ("4","6","12")

You can read more about the dynamic and iterate methods in the iBatis documentation.

Retrieving complex objects from a database using iBatis

2010-02-27T18:00:00.000-08:00

We have been using iBatis for 2 years now on the project I am on. We had need to retrieve a complex object from the database.
Basically, we have an object obA that has some fields(string, int, ..., and one that is List where obB has some fields(String, int, date, ...). Instead of retrieving all of the obA objects then looping through each of those to get the array of obB objects for each, I wanted to do it in one shot.

A little digging and I found that iBatis supports this and it really wasn't that hard. This simple example will generate retrieve an array of Company objects that each contain an array of Employee objects
.
First here are the two Java Classes for Company and Employee. To simplify, imports, getter/setter methods, etc have been omitted.


package com.projectA;
public class Company {
    private int id;
    private String name;
    private String address;
    private String city;
    private String state;
    private String zipCode;
    private List employees;
}

package com.projectA
public class Employee {
    private int companyId;
    private int id;
    private String firstName;
    private String lastName;
    private String phoneNumber;
    private String ssn;
    private Date dob;
}

The following is the iBatis XML file that will retrieve a list of all companies sorted by company name, each containing a list of employees sorted alphabetically by last name, then first name.


<sqlMap namespace="Company">

    <typeAlias alias="company" type="com.projectA.Company"/>
    <typeAlias alias="employee" type="com.projectA.Employee"/>

    <resultMap id="companyResultMap" class="company">
        <result column="company_id" property="id" />
        <result column="company_name" property="name" />
        <result column="company_address" property="address" />
        <result column="company_city" property="city" />
        <result column="company_state" property="state" />
        <result column="company_zip_code" property="zipCode" />
        <result column="{companyId=company_id}" property="employees"
                   select="Company.getEmployees"/>
    </resultMap>

    <resultMap id="employeeResultMap" class="employee">
        <result column="employee_company_id" property="companyId" />
        <result column="employee_id" property="id" />
        <result column="employee_first_name" property="firstName" />
        <result column="employee_last_name" property="lastName" />
        <result column="employee_phone_number" property="phoneNumber" />
        <result column="employee_ssn" property="ssn" />
        <result column="employee_dob" property="dob" />
    </resultMap>

    <sql id="select_company_fragment">
        SELECT company_id,
               company_name,
               company_address,
               company_city,
               company_state,
               company_zip_code
        FROM company
    </sql>

    <sql id="select_employee_fragment">
        SELECT employee_company_id,
               employee_id,
               employee_first_name,
               employee_last_name,
               employee_phone_number
               employee_ssn,
               employee_dob
        FROM employee
    </sql>

    <select id="getAllCompanies" 
               resultMap="companyResultMap">
        <include refid="select_company_fragment"/>
        ORDER BY company_name
    </select>

    <select id="getEmployees" 
               resultMap="employeeResultMap">
        <include refid="select_employee_fragment"/>
        WHERE employee_company_id=#companyId#
        ORDER BY employee_last_name, employee_first_name
    </select>

</sqlMap>

That's all there is to it. So when getAllCompanies is called, it will automatically call the getEmployees method to fill in the list of employees for each company. You can see this by looking at the companyResultMap, it has a column called employees that is mapped to a select statement and passes the current company id into that select.

Some benefits are simplicity and performance. One dao call retrieves everything I want, instead of getting a list of companies and needing to iterate over each to get the list of employees.

A nice ibatis feature is the alias. The benefit of using the alias in the ibatis XML file is that instead of needing to put com.projectA.Company in multiple places in the file, you do it once and reference it. Then if the package structure is refactored, you only have to change it once.

Another nice ibatis feature is the sql sections. This allows you to have SQL fragments, that can be used by more than one select. So in our example if you wanted to select all the companies for a given city you could add the following:


    <select id="getAllCompanies" 
               parameterClass="String" 
               resultMap="companyResultMap">
        <include refid="select_company_fragment"/>
        WHERE company_city=#city#
        ORDER BY company_name
    </select>

This is the same as the select statement above, except it has the additional constraint on the city.

More iBatis posts to follow.

JSON Response Encoding Troubles

2009-07-29T13:33:00.001-07:00

On the project I am currently on, we have had some troubles with some of the JSON responses returned from the server. Specifically if the strings contained escaped characters. We are using ExtJs on this project and upon receiving the response on the client, we were doing an Ext.decode(response.responseText); This was throwing exceptions, like error: unterminated string literal.

All the values that caused problems were escaped characters.
\"
\\
\r
\n
\t

It turns out that those escape sequences in the response are treated as part of the JSON source, not the JSON data. If your data contains escaped characters, the '\' needs to be encoded so that it becomes part of the data, and the Ext.decode(or eval) will then remove the extra '\'. The list above becomes:

\" --> \\\"
\\ --> \\\\
\r --> \\r
\n --> \\n
\t --> \\t

\\ is particularly bad if it is the last character in the data and it is not encoded, as it will combine with the field terminating ", and cause the JSON to fail.

\n or \r, cause the JSON source to have a return in the middle of the JSON, thus invalidating it.

\" will cause the field to prematurely terminate, causing the JSON to fail.

\t is really pretty harmless, as it actually tabs the data, but at least it doesn't cause the JSON to fail. Other ones that may be of concern, but haven't investigated.
\b - backspace
\f - formfeed
\v - vertical tab
\' - single quote(haven't noticed this one being a problem)

Grid Panel as a form panel Element

2009-04-13T11:59:00.000-07:00

We recently ran into a little problem in that we needed a grid panel to be displayed on a form panel. It displayed fine but it was left justified and made the form look pretty bad. One of my co-workers stumbled on to an undocumented feature that solves the problem. I just wanted to share it and post it for my own future reference.

For the life of us, we couldn't figure out how to tell the GridPanel to render in the fieldItem column and not the fieldLabel (All the other items let you define a fieldLabel property, but the GridPanel didn't have this option) column of the FormPanel. We finally found out that there is a property that is not documented anywhere, or anywhere that I can find, that tells the GridPanel that it is a formItem. So, in order to treat a GridPanel like a formItem all you have to do is set a property "isFormField" to true, slick huh. Now if you want to, you can also define a fieldLabel for that as well. This will align your GridPanel with all your other fieldItems.

Hope this is helpful.

Manipulating Arrays in JavaScript

2009-02-25T13:30:00.001-08:00

In my recent work I have had to do a lot of array manipulation in JavaScript. While doing this I learned a few tricks, that I did not want to forget and thought others might benefit from.

Some of these things are simple, but I include here for completeness. I welcome any feedback on any array tricks that you have learned.

Adding to an array
```
var array = [];
array.push(5835);
```

Deleting from an array
```
var array =[];
array.push(5835); // index of 0
array.push(6835); // index of 1
array.push(7835); // index of 2
delete array[1];
```
The resulting array would be [5835, 7835], but the indexes would still be [0,2]. JavaScript does not reorder the indexes to always be zero based.

Array Looping
Although you can loop through an array normally:
```
for(var i = 0; i < array.length; i++) {
  alert("Index was " + i + ", value was " + array[i]);
} 
```
You can get into trouble, if you are doing array manipulation like deleting. Like I stated in #2, there is no more index 1, so you would get a JavaScript error.
Because of this I have become a fan of the following loop:
```
for(var i in array) {
  if (typeof array[i] == "function") {
    continue;
  }
  alert("Index was " + i + ", value was " + array[i]);
}
```
A couple of things to note, first is the check to make sure the element is not a function. All arrays have a remove method inherent in them, so you need to skip that. Second, you will see that the indexes are indeed, 0, 2 and not 0, 1, if you use the dataset from #2.

More to come later...

Dynamic Size Arrays in javascript

2008-10-27T13:11:00.000-07:00

Recently I needed to create a dynamic array in javascript. I didn't want to have two complete arrays defined and choose based off an if statement. So to do this I used the push() command. It easily allowed me to tailor the contents of the array based on various boolean flags. The array I was creating was for the tools param of a new Ext.Panel object. As you can see in the sample code below, the publish button is only added to the tool array if the isShareable flag is true.

var tools = [];

tools.push({
id:'toggle',
handler: this.onToggle.createDelegate(this),
qtip: "toggle"
});
tools.push({
id:'help',
handler: this.onHelp.createDelegate(this),
qtip: "help"
});
tools.push({
id:'gear',
handler: this.onConfigure.createDelegate(this),
qtip: "configure"
});
tools.push({
id:'refresh',
handler: this.onRefresh.createDelegate(this),
qtip: "refresh"
});
if (this.isShareable) {
tools.push({
id:'save',
handler: this.onPublish.createDelegate(this),
qtip: "publish"
});
}
tools.push({
id:'close',
handler: this.onClose.createDelegate(this),
qtip: "close"
});

NoClassDefFound Exceptions tied to PermGen Space

2008-09-22T14:47:00.000-07:00

Today I spent a good deal of time trying to figure out why I was getting a bunch of NoClassDefFoundErrors. The changes I made were not significant, but I did make one configuration change to JBoss. It seems the default is to compile the JSP pages with Java 1.4, so I changed it to 1.5, as a new JSP page used some Java 1.5 generics.

When I tried to bring the application back up, I got the following different errors each time I restarted JBoss.

javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/problem/DefaultProblem

java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/ast/Argument

NoClassDefFoundError: org/eclipse/jdt/internal/compiler/ast/SingleTypeReference

Finally, I got a PermGen space error.

A friend suggested increasing my PermGen memory in JBoss, which I did by adding the following line to the run.bat file in the JBoss bin directory:
set JAVA_OPTS=%JAVA_OPTS% -XX:PermSize=512m -XX:MaxPermSize=512m

This fixed the problem. So when having weird org/eclipse NoClassDefFoundErrors being thrown, try increasing your PermSize.

Custom Events in ExtJS

2008-09-04T13:58:00.000-07:00

I was kind of new to the idea of event driven applications, but after reading a little about it fell in love with it. I immediately had some applications in my work where it would solve particular problems I was having. Having never implemented this before, I didn't know where to start. Searching the ExtJS forum I uncovered some good information, but not enough to just jump right into it.

I thought I would blog about it while it was still fresh in my mind.

First you need to extend the Observable class for the class that you want to add a new custom event to. Then in a method called when the class is constructed, you need to add the new custom event. I needed to throw a custom event whenever the tab changed. As you can see from the code below, that a 'tabchange' event triggered a call to onTabChange function. This function then fires the 'myCustomEvent' event. That is all it takes to create a custom event. Of course, nothing will happen if there are not any listeners registered for that event.

Example.Events = function(config) {
this.initialize(config);
};
Ext.extend(Example.Events, Ext.util.Observable, {
initialize: function(config) {
this.addEvents('myCustomEvent');
this.panel = new Ext.TabPanel({
.....
};
this.panel.on('tabchange', this.onTabChange.createDelegate(this));

}
onTabChange : function(tabPanel, tab) {
var index = -1;
if (tab.getIndex) {
index = tab.getIndex();
}
this.fireEvent('myCustomEvent', index);
},
}

In my case, I had widgets that existed on tabs and they need to know when their tab was activated. So inside those widgets they needed to obtain a reference to the Example.Events object for our application, and then register as a listener and tell it which function to call back on. I did this in the widget constructor, as shown below.

in constructor:

var event= Example.Parent.getEvent();
event.on('myCustomEvent', this.tabSelected.createDelegate(this));
this.myTabIndex = event.getMyIndex();

The call back function:
tabSelected : function(index) {
if (index == myTabIndex) {
// Do something if your tab is activated
}
},

That's it. There is nothing more to it. And it works like a charm.

JavaScript handling of time zones

2008-09-04T07:53:00.000-07:00

On the project I am currently working on, we had a problem to solve regarding time zones with our application. From the web interface the user was entering time for some of the fields. The time was being sent to the server and stored in a database. The problem we ran into was the client time zone was different than the server timezone. The database on the server was converting all times that were saved to the server time zone. When the data was retrieved, it was displaying the server time values and presenting it to the user as if it were client local time. Obviously we were not taking time zones into account.

Like most problems, there is usually more than one way to solve it. The way we solved it, was to make sure that all of the times that were being passed back and forth through the REST web service calls were in GMT (Zulu) time. The database can convert to local time, but before it is sent to the client, it is converted back to GMT. Likewise, in the browser, the user can enter time in local time, but before it is sent to the server it is converted to GMT time.

The rub came in finding the right java script date functions to allow for conversion to and from GMT. The GMT string representations that were being passed back and forth, needed converted to the clients local time. This was easily accomplished by using the java script Date constructor.

var localTimeZoneDate = new Date(gmtTimeZoneDateStr);

The date components the UI implemented, needed to be initialized with milliseconds from epoch time. To get that information we simply called Date.parse(localTimeZoneDate);

On the server it was equally easy to convert the value retrieved from the database to GMT.
String dateFormatStr = "EEE, dd MMM yyyy HH:mm:ss z";
SimpleDateFormat dateFormat = new SimpleDateFormat(dateFormatStr );
dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
Date date = dateFormat.parse(timeStr);

In reviewing what we did, there may have been an easier solution. Both the Java Date object and the JavaScript Date object, are able to return milliseconds from epoch. It would have been much simpler to just send that value back and forth, and not worry about the string format.

How to add new events types to ExtJS components

2008-04-29T10:34:00.000-07:00

I needed to be able to click on an Ext.Panel and have a window popup. The 'click' event is not a supported event for Ext.Panel. It turns out it is pretty easy to do in ExtJS. Almost every component has a getEl() method.

var myPanel = new Ext.Panel({
html: '<> Click Me < / h2>',
autoHeight: true
});

Once the panel has been rendered you simply, need to do the following.
var element = myPanel.getEl();
element.on('click', this.handleClick.createDelegate(this));

Now when ever a mouse clicks on the panel, the handleClick method will be called. The trick is that you have to wait until the panel has been rendered, to be able to add the event to it.

You may be tempted to use
myPanel.on({'click':this.handleClick.createDelegate(this), scope: this});

This method works great for events that the ExtJS Component supports by default, but doesn't work for other events.

How to add a keymap to an existing FormPanel in ExtJS

2008-04-29T10:15:00.000-07:00

Recently I needed to add a key map to an existing FormPanel in our ExtJS project. This sounded like a simple thing to do and in the end the code was simple, but it wasn't obvious how to do it. So I thought I would try to save you the pain I went through to figure it out.

In the JavaScript, you might have something like the following.

this.myForm = new Ext.form.FormPanel({ ...your parameters....});
this.myWindow = new Ext.Window({ ...your parameters....
items: this.myForm,
buttons: [{
text: 'Save',
handler: this.onSave.createDelegate(this)
},{
text: 'Cancel',
handler: this.onCancel.createDelegate(this)
}]
});

Now how do you go back and add a KeyMap after the fact.

The panel has to be rendered before you can add the KeyMap to it. To do that simple call the following.

this.myWindow.show();
this.myWindow.doLayout();

Now you can create the KeyMap.
new Ext.KeyMap(this.myForm.body, [{
key: Ext.EventObject.ESC,
fn: this.onCancel.createDelegate(this),
scope: this
},{
key: Ext.EventObject.ENTER,
fn: this.onSave.createDelegate(this),
scope: this
}]);

Now, the user can either click the cancel button or hit 'esc' to cancel what he was doing and to save what he was doing he can either press the save button or press the 'enter' key on the keyboard. I hope you find this helpful.

Debugging JavaScript in IE

2008-04-25T09:28:00.000-07:00

Firebug is so powerful and easy to use, I almost never need to debug JavaScript in IE. However, every once in a while I will come across a bug that only occurs in IE. A co-worker of mine told me about the best free IE JavaScript debugger he was aware of. It has been a real life saver at times. I thought I would share the information here so that you too can debug JavaScript in IE, if you must.

It does take a while to download and install the Mircosoft Visual Web Developer 2008 Express Edition, http://msdn.microsoft.com/vstudio/express/vwd/

This blog entry has steps on how to set up and use the tool the first time.
http://www.berniecode.com/blog/2007/03/08/how-to-debug-javascript-with-visual-web-developer-express/

There some special instructions in that blog for getting it to work, if IE is not your default browser.

Hope you find this helpful

Configuration Management: Bugs and Branches

2008-04-11T10:00:00.000-07:00

Most software when the time comes to make a release will establish a branch. As the branch is tested bugs will be found that must be fixed. A very common dilemma is do we fix it on the main branch then merge the fix to the release branch, or do we fix it in the release branch and merge it back to the main branch. Either way, the changes must be in both locations.

This is a very old debate, and developers will usually be adamant about which side they support. I have been involved in projects that have practiced both methods. I would simply like to share my experiences from both sides.

Branch to Main:
Pros

One of the main rationals for this method is that the release is on a time line to get out. The fixes need to be put into the branch first to be able to maintain schedule and insure that the problems are fixed to support the customer.
Mainline code will continue to be enhanced for future releases so it will diverge from the branch code and the fix for the main branch may not be valid for the branch, so you should fix the branch first.

Cons

The biggest risk is that the fix never makes it back to the main branch so the next release you risk breaking the customer. Or even worse, an enhancement that was requested at the last minute by the customer, never makes it back to the main line.

Main to Branch:
Pros

It guarantees that the main branch always has all the fixes and enhancements, so that future releases are not at risk of missing critical bug fixes or minor customer enhancements.

Cons

It takes longer to update the branch with fixes, potentially causing a delay in the software release.
Risk that the fixes from the main branch don't get merged to the release branch.
The longer the branch lives the more difficult the merges to the branch become. You only want the bug fixes, not the new development.

Experiences:
My experiences have shown me that the testing involved on the release branch is much more rigorous than the testing on the main branch. No one wants the software to fail for the customer so must of the testing is focused on the release. Due to new development, the main branch is "broken" more frequently and while that is not desired, it is not entirely unexpected. It is more difficult to test reliably the main branch because the development my not be finished, and bugs would be written against code that wasn't ready to be tested. The release branch is usually much more stable with only critical defects being fixed, which makes it easier to test.

I have been involved in releases of software that went out to the customer only to have bugs reported back, that were fixed in a previous version and are broken again in the version they just received. Customers are not very understanding of this situation, especially on government contracts. They will call into question the processes you use, the reliability of your release process, even your ability to write bug free code. Some very uncomfortable meetings to have to go through.

One project I was on, the branch lived for a year and a half. The mainline code had diverged so much that bug fixes were completely different in both places. In fact we branched 3 times from the original branch without ever going back to the main. It was truly a nightmare to maintain. Which ever method you choose, the longer the branch lives, the more difficult the merges become, either direction.

There are pros and cons to both sides. From my experience, release branches should be short lived, and bug fixes should be merged in as quickly as possible after you branch. This negates a lot of the code divergence cons used as arguments for both sides. Due to the heavy testing on the release branch, it is not very likely that a required bug fix will be missed on the branch.

For the reasons stated above I am a proponent of fixing in the main branch then merging to the release branch. That said both methods can work. Going from the branch to the main requires a well defined process that everyone must follow to guarantee that changes are merged into main once they are verified on the branch.

Getting Internationalization to work on Windows

2008-04-10T16:00:00.000-07:00

Recently, I wasted several days trying to resolve an internationalization issue within our web application. Now I would like to share my experience and the solution I found in order to save others time.

Our web application has a requirement to support multiple languages, English and Hangul(Korean). Our application uses ExtJS for the front end and Jersey for the REST services, and we had difficulties getting i18n working in the javascript.

Initially all the Korean characters would appear as ??? in our ExtJS widgets. After some research we added a call to get the UTF-8 bytes, new String(bundle.getValue(key).getBytes(“UTF-8”)). and setting the content type charset to UTF-8 on the Jersey StringRepresentation. The issue with this approach was that certain Korean characters would appear in our ExtJS widgets as ?? and others would appear fine, so it was only a partial solution.

The solution ended up being pretty simple. Based on some information here, you basically have to set the default charset for the JVM to UTF-8. The default charset on Windows is windows-1252. To do this set the system property file.encoding=UTF-8 in the JVM, for JBoss you can do this by modifying the run.bat and adding -Dfile.encoding=UTF-8 to the JAVA_OPTS variable. Please note this is a Windows specific issue, since UTF-8 is the default charset for both Linux and Macs.

The following is the code before and after the changes to the JVM default charset , that will allow you to properly visualize Hangul(Korean) in your application.

The following displayed ?? for certain Korean characters.
ResourceBundle bundle = ResourceBundle.getBundle(bundleName, locale);
Enumeration keys = bundle.getKeys();
String key = keys.nextElement();
String value = bundle.getValue(key);
byte[] bytes = value.getBytes("UTF-8");
String newValue = new String(bytes);
StringRepresentation representaiton = new StringRepresentation(newValue);
representaiton.setLanguage("ko");
representaiton.setMediaType("text/plain; charset=UTF-8");
return representation;

After configuring the JVM to set the default charset to UTF-8
ResourceBundle bundle = ResourceBundle.getBundle(bundleName, locale);
Enumeration keys = bundle.getKeys();
String key = keys.nextElement();
String value = bundle.getValue(key);
StringRepresentation representaiton = new StringRepresentation(value);
representaiton.setLanguage("ko");
representaiton.setMediaType("text/plain; charset=UTF-8");
return representation;

For those that want some additional details as to why it wasn't working, read on.
bundle.getValue(key) always returns a UTF-8 String, even if the default charset is windows-1252.

The Jersey REST Service is returning a JSON StringRepresentation back to the client code. But the StringRepresentation class expects the string you construct it with to be in the default charset . Setting the media type on the StringRepresentation to "text/plain; charset=UTF-8" will cause it to try to convert the string it was provided from the default charset to UTF-8.

The first case it was trying to convert a UTF-8 String(thinking it was windows-1252) to a UTF-8 String. This produces all ?????? on the client for the Hangul(Korean) characters.

The second case, we get take the UTF-8 string returned from the bundle and convert it to an array of UTF-8 byte codes. We then take those byte codes and create a new windows-1252 string. The string representation class basically takes that windows-1252 string and converts it back to a UTF-8 string. So why didn't this work? Some byte codes can't be represented in windows-1252, specifically(129, 141, 143, 144, and 157). So any Korean characters that had one of those specific byte codes, would show up as a '?'.

The final solution, eliminates all the extra conversions. Bundle returns a UTF-8 string, StringRepresentation sees that the default charset is UTF-8, and it is sending UTF-8 so it doesn't need to convert anything and the result is the client correctly displays all the Korean characters.

In summary, just set the default charset on the JVM to UTF-8.

To find out what default charset your system is set to you can execute the following Java command:
System.out.println(java.nio.charset.Charset.defaultCharset().name());