Exploring Spring Integration & Spring Web Service, Part 1

I have been learning Spring Integration and Spring Web Services on the project I am on. The more I learn the more I like the framework, but I have to say it has not been easy to make progress. The number of resources available is lacking. I thought I would spend the next several blogs covering some of the information I have been learning.

I needed to do was expose four SOAP Web Services based on a set of WSDL and XSD files provided by the vendor. First thing that had to happen was setting up the web.xml file. From reading the documentation on the main Spring site, I determined to use MessageDispatcherServlet.

web.xml

<web-app>
   <display-name>My Application Name</display-name>
   <servlet>
       <servlet-name>si-ws-gateway</servlet-name>
       <servlet-class>org.springframework.ws.transport.http.MessageDispatcherServlet</servlet-class>
       <init-param>
           <param-name>contextConfigLocation</param-name>
           <param-value>classpath:si-ws-gateway-config.xml</param-value>
       </init-param>
       <load-on-startup>1</load-on-startup>
   </servlet>
   <servlet-mapping>
       <servlet-name>si-ws-gateway</servlet-name>
       <url-pattern>/*</url-pattern>    
   </servlet-mapping>
</web-app>

Next, let's take a look at the spring application context file.
si-ws-gateway-config.xml

    <bean class="org.springframework.ws.server.endpoint.mapping.UriEndpointMapping">
        <property name="mappings">
            <props>
                <prop key="${data.subscriber.url}">dataGateway</prop>
                <prop key="${request.status.url}">statusGateway</prop>
        </property>
    </bean>

    <si:channel id="dataArrivedChannel" />
    <si-ws:inbound-gateway id="dataGateway" request-channel="dataArrivedChannel" />
    <stream:stdout-channel-adapter id="stdoutAdaptorWithDefaultCharsetDataArrived" channel="dataArrivedChannel"/>

    <si:channel id="statusChangedChannel" />
    <si-ws:inbound-gateway id="statusChangedGateway"
                           request-channel="statusChangedChannel"
                           marshaller="marshaller"
                           unmarshaller="marshaller"/>
    <si:service-activator input-channel="statusChangedChannel"
                          ref="statusServiceEndpoint"
                          method="processStatusMessage"/>   

    <bean id="marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>com.ourvendor.StatusUpdate</value>
            </list>
        </property>
    </bean>

    <bean id="StatusDao" class="com.company.StatusDao"/>

    <bean id="statusServiceEndpoint"
          class="com.company.StatusServiceEndpoint">
        <constructor-arg ref="statusManager"/>
    </bean>

    <bean id="statusManager"
          class="com.company.StatusManagerImpl">
        <constructor-arg ref="statusDao"/>
    </bean>

Well that seems like a lot of spring configuration, but it eliminates a bunch of Java code. Let's break down this configuration a bit. First, the UriEndpointMapping bean. You might notice that there are some values provided from a properties file. I will be covering properties in more detail in a future blog. For now, just know spring replaces the ${data.subscriber.url} and ${request.status.url} with a valid URL value from a properties file. This defines which URLs will be routed to the specified spring integration web services inbound-gateway. The gateway will place the SOAP message onto the specified channel as a DOMSource object.

As seen above the dataGateway does not specify a marshaller and unmarshaller which is why the DOMSource object is placed on the channel. The statusChangedGateway does specify the marshaller and unmarshaller, so when it places the object on the channel, it is not a DOMSource object, but a StatusUpdate object. As you can see it is very easy to leave the input in it's native format or convert it to a Java Object. In one case the object was only going to be routed(eventually) to a JMS queue, and did not need to be unmarshalled, in the other case it required unmarshalling so that a service activator endpoint could save the object to the database. As we will see in a future blog, it would have been nice if there was a way to specify the output to the channel as a string, instead of DOMSource. This would be a great enhancement to spring integration.

The service activator is an endpoint that allows a specific method to be called, and passes the object from the incoming channel to the method. It can also place the object, either unmodified or modified, onto an outgoing channel to be further processed by the system. The rest of the configuration is very straightforward spring beans with dependency injection. This works very well if each of the web services have a unique URL, but what happens if all calls come into a single URL. That will be the topic of the next blog in this series.

Accessing External Properties Files in WebLogic

It is not only common to do, but is also a very good practice to put values that may need to change into a properties file. This creates a central location to modify values. Especially for the configuration a system based on the installation location, etc. Having the properties file inside the .war is very inconvenient when you want to change it as it causes you to recompile your code, even if it hasn't changed. I was on a project that was able to pull the property file out of the .war and put it in the JBoss AS conf directory. The application could still access it. Unfortunately the project I am on now uses WebLogic. After lots of google searching and asking a few co-workers if they new how to do it, I did find the answer.

Simply place the properties file in $WEBLOGICHOME/user_projects/domains/mydomain where mydomain is the name of he domain your app uses.

Works great and now I can change properties all day long without recompiling.

Remote Desktop & Change Password

I ran into a problem where I needed to change the password on an account that was on a server half way around the world. All I could do was Remote Desktop in to that box, and when I did Ctrl + Alt + Delete, it was changing the password on the box I was physically at, not the one in the RD session. After some searching I found this solution: Ctrl + Alt + End.

Hope this helps someone.

Java JDBC Performance vs. iBatis

While working on a recent project, I worked with a database that contained 23 billion rows in one table, and several hundred million in another table. Performance retrieving information was critical for success. I have used Spring and iBatis for several years now and only had a couple of weeks to develop a prototype, so implemented the database access layer using iBatis and spring. It is quick to implement and easy to maintain. We had several use cases we wanted to test. Some use cases returned as few as one row, some returned as many as 20 million. Unfortunately, it was taking too much time to retrieve the information for most of the use cases. One downside of using iBatis, was that I couldn't determine if the time was spent in the database query, returning the results to Java, or in creating the java objects themselves. I decided to modify the code to use straight JDBC calls so that way I could time the retrieval separately from converting to Java Objects.

I did not expect to see much performance improvement converting to straight JDBC. However, just converting to straight JDBC was a substantial performance improvement over using iBatis(30-40% faster). Unfortunately this still wasn't enough. This was a web application being deployed to WebLogic. After some research a co-worker found that the default number of rows retrieved by the WebLogic data source was 10. Queries whose results were less than 10, performed very well, but most queries returned thousands or millions of rows, so performance was very poor. We initial bumped that value to 1000, and saw another 40+% performance improvement. This in turn prompted some investigation of JDBC, and I found that the default value for JDBC against an oracle database was 10. Inside the code I increased the fetch rows value to 1000, and saw another 30% performance improvement. I concluded that more was better on the fetch rows setting, so I increased it to 50,000 and found that I now had a new problem: running out of memory. After some additional modifications to the memory available to the web application, and reducing the fetch count to 5000, I was able to get a reliable result. Essentially, a higher fetch count, meant less database connections to retrieve the data.

Using iBatis is a great way to get a lot done in a short amount of time, and for smaller databases I still prefer to use it. In addition to being easy to implement, it is also easy to maintain. However, if you have a large database, then using Straight JDBC might be a good solution. Don't forget to take fetch size into account both in the application server and in the Java JDBC code when dealing with large databases. The larger the fetch size, the fewer database connections that will be used, but the more memory that will be required. I did not investigate setting the fetch size in iBatis, so do not know if that is possible.

In the Java JDBC code setting the fetch size is done as follows:

            String query = "SELECT * FROM my_table";
            Connection conn = _dataSource.getConnection();            
            PreparedStatement st = conn.prepareStatement(query);
            st.setFetchSize(5000);
            ResultSet rs = st.executeQuery();

Some other Java improvements that helped with performance, was reducing the number of java objects being created and destroyed. This includes making any strings that are reused static member variables so they are only created once and reducing the number of java objects that are being created. If you are outputting to a file or or a message, perform those operations inline to reduce creating intermediate objects.