Spring Integration Marshalling with JAXB

The project I was working on used SOAP based Web Services. The .wsdl and .xsd files that described the services we needed to integrate with were provided by the vendor. Often times I needed to access information in the SOAP responses, or I needed to easily build a SOAP request object. While you can work directly with the XML, it is much easier to use marshalling/unmarshalling techniques. Being a proof of concept I chose JAXB to do the marshalling/unmarshalling. There may be better choices depending on your needs and most likely spring integration supports it.

While it is possible to create by hand JAXB Java classes, it seems to make more sense to use the xjc tool and generate the classes automatically. I will not go into great detail on using this tool as there are other resources that do that. xjc was on my class path, so I was able to navigate to the directory containing the .xsd and .wsdl files. I created a directory there called temp and ran the following command on each of the files that I needed classes generated for, which would create all the classes I needed in the appropriate directory structure under temp.

xjc -d temp myVendorFile.xsd

Now that the files had been generated it was a matter of configuring spring to use them. There are several ways to go about this. You can create a generic marshaller that handles all classes, or you can create more targeted marshallers. If you are creating a generic one you might just set the id to marshaller, if you want more targeted the id name will reflect something more specific.

    <bean id="marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>com.vendor.JAXBClass1</value>>
                <value>com.vendor.JAXBClass2</value>
                <value>com.vendor.JAXBClass3</value>
            </list>
        </property>
    </bean>

    <bean id="myServiceClient" class="com.company.project.ws.MyServiceClient">
        <property name="marshaller" ref="marshaller"/>
        <property name="unmarshaller" ref="marshaller"/>
    </bean>

the class MyServiceClient extends the spring WebServiceGatewaySupport class. This is an easy way to send requests and receive responses. Inside your class you can create the request object that you want sent using the JAXB ObjectFactory methods.

MyRequest myRequest = new ObjectFactory().createMyRequest(); 

Then you can set individual elements on that object, whatever they are. The request is sent by using a method from the Spring class. The SoapAction is not set automatically, so it needs to be set manually.

MyResponse myResponse = getWebServiceTemplate().marshallSendAndReceive(serviceUri, myRequest, new WebServiceMessageCallback() { public void doWithMessage(WebServiceMessage message) {
((SoapMessage)message).setSoapAction("MySoapAction");}});

That's really all there is to it. As you can see the above method automatically marshalls the request to XML and unmarshalls the response to the JAXB Java object. Also it is good to surround that method call with a try catch so that if there is a problem on the other end, your software can react appropriately.

Using Variable Substitution from Configuration Files in Spring

It doesn't take long in the development world to learn that hard coding property values is a bad idea.  Hard coded values will require the software to be rebuilt and redeployed if a change is required.   Moving values to configuration files so they can be modified in a central location is a first step.  A second step is moving the configuration file outside of the .war or .ear, so they can be modified at run time to avoid recompiling.  To read about how to externalize property files from your application see my earlier blog.

There are several places in the Spring configuration that can make use of variable substitution from a property file.  First things first. If you create a property file how do you access it from your spring configuration file?  Near the top of the configuration file add the following bean definition:

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
         <property name="locations">
            <value>file:my-project.properties</value>
         </property>
    </bean>

If you need to have comments in the property file, start the line with a #.  Properties should be entered into the property file in the following format: property.name=property value, i.e.

my.favorite.search.engine=http://www.google.com

Properties can be referenced from value fields, as well as bean variables.

	<bean id="jmsQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
		<property name="jndiName" value="${my.project.testQueue}"/>
		<property name="jndiTemplate" ref="jndiTemplate" />
	</bean>
	<si-ws:outbound-gateway id="WebServiceGateway"
			 request-channel="jmsChannel"
			 reply-channel="replyChannel"
			 uri="${my.project.uri}"  />

It is also possible to access properties from spring annotations which can be very useful. The scheduled annotation is a good example.

@Scheduled(cron="${my.project.schedule.cron}")

Spring makes it very easy to leverage properties files for making systems easier to configure on the fly. There is no need to continue hard coding values that you or your client may want to change.

Configuring Web Logic 10.3.5 and log4j for your Java Application

I recently spent several days trying to track this down why Apache log4j for my Java application was working when deployed to Oracle WebLogic Server 10.3.5. I am posting my findings here in the hopes of saving others some time. I found several blogs that talked about this, but the end result was that log4j was still not working correctly. There was always one step missing. A co-worker and I finally figured it out for both Linux and Windows.

1. Copy wllog4j.jar from your WebLogic server/lib directory and place it in your domain_root/lib folder.
2. Copy your version of log4j.jar to your domain_root/lib folder.
3. Copy your log4j.xml to your domain_root folder.
4. Log in to the your WebLogic admin server console. Click on Servers -> Admin Server -> Logging. Click on advanced mode, and change the logging implementation from JDK to Log4J. Save your changes.

Most of the blogs had these items, but the one critical piece that was missing:
5. (Linux)Edit the setDomainEnv.sh file in the domain_root/bin directory. and find the following code:

if [ "${LOG4J_CONFIG_FILE}" != "" ] ; then
       JAVA_PROPERTIES="${JAVA_PROPERTIES} -Dlog4j.configuration=file:${LOG4J_CONFIG_FILE}"
       export JAVA_PROPERTIES
fi

Insert the following above those lines of code(replacing the path with the one for your system's domain_root directory:

LOG4J_CONFIG_FILE=”/u01/app/oracle/middleware/user_projects/domains/base_domain/log4j.xml”
Export LOG4J_CONFIG_FILE

5. (Windows)
Edit the setDomainEnv.cmd file in the domain_root/bin directory. and find the following code:

if NOT "%LOG4J_CONFIG_FILE%"=="" (                                                          
            set JAVA_PROPERTIES=%JAVA_PROPERTIES% -Dlog4j.configuration=file:%LOG4J_CONFIG_FILE%
)

Insert the following above those lines of code(replacing the path with the one for your system's domain_root directory:

set LOG4J_CONFIG_FILE=”C:\Oracle\Middleware\user_projects\domains\base_domain\log4j.xml”

6. Activate the changes and restart the admin server.

Here is a sample log4j.xml file. Obviously, you will need to update it for a valid location for your file appender, and replace yourcompany and yourproject with valid values.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
   <!--====================================================================-->
   <!--    Appenders     -->
   <!-- %d{dd-MMM-yy HH:mm:ss} %-5p [%c{1}] %m%n  -->
    <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="[%p] - %m (%F:%M:%L)%n"/>
            <!--param name="ConversionPattern" value="%d{ISO8601} %p [%c{1}] - %m (%F:%M:%L)%n"/-->
        </layout>
    </appender>

    <appender name="FILE" class="org.apache.log4j.RollingFileAppender">
        <param name="File" value="/temp/logs/yourproject.log"/>
        <param name="Append" value="true"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ISO8601} %t %-5p %c{2} - %m%n"/>
        </layout>
    </appender>

    <!--====================================================================-->
    <!--    Logging Levels     -->

    <category name="com.yourcompany.yourproject">
      <priority value="DEBUG"/>
    </category>

    <category name="org.apache">
        <priority value="WARN"/>
    </category>

    <category name="org.springframework">
        <priority value="WARN"/>
    </category>

    <!--===================================================================-->
    <!--    Appender Assignments     -->
    <root>
        <priority value="ERROR"/>
        <appender-ref ref="FILE"/>
    </root>
</log4j:configuration>

Again, I hope that this will save someone the pain I went through to track this down.

Exploring Spring Integration & Spring Web Service, Part 1

I have been learning Spring Integration and Spring Web Services on the project I am on. The more I learn the more I like the framework, but I have to say it has not been easy to make progress. The number of resources available is lacking. I thought I would spend the next several blogs covering some of the information I have been learning.

I needed to do was expose four SOAP Web Services based on a set of WSDL and XSD files provided by the vendor. First thing that had to happen was setting up the web.xml file. From reading the documentation on the main Spring site, I determined to use MessageDispatcherServlet.

web.xml

<web-app>
   <display-name>My Application Name</display-name>
   <servlet>
       <servlet-name>si-ws-gateway</servlet-name>
       <servlet-class>org.springframework.ws.transport.http.MessageDispatcherServlet</servlet-class>
       <init-param>
           <param-name>contextConfigLocation</param-name>
           <param-value>classpath:si-ws-gateway-config.xml</param-value>
       </init-param>
       <load-on-startup>1</load-on-startup>
   </servlet>
   <servlet-mapping>
       <servlet-name>si-ws-gateway</servlet-name>
       <url-pattern>/*</url-pattern>    
   </servlet-mapping>
</web-app>

Next, let's take a look at the spring application context file.
si-ws-gateway-config.xml

    <bean class="org.springframework.ws.server.endpoint.mapping.UriEndpointMapping">
        <property name="mappings">
            <props>
                <prop key="${data.subscriber.url}">dataGateway</prop>
                <prop key="${request.status.url}">statusGateway</prop>
        </property>
    </bean>

    <si:channel id="dataArrivedChannel" />
    <si-ws:inbound-gateway id="dataGateway" request-channel="dataArrivedChannel" />
    <stream:stdout-channel-adapter id="stdoutAdaptorWithDefaultCharsetDataArrived" channel="dataArrivedChannel"/>

    <si:channel id="statusChangedChannel" />
    <si-ws:inbound-gateway id="statusChangedGateway"
                           request-channel="statusChangedChannel"
                           marshaller="marshaller"
                           unmarshaller="marshaller"/>
    <si:service-activator input-channel="statusChangedChannel"
                          ref="statusServiceEndpoint"
                          method="processStatusMessage"/>   

    <bean id="marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>com.ourvendor.StatusUpdate</value>
            </list>
        </property>
    </bean>

    <bean id="StatusDao" class="com.company.StatusDao"/>

    <bean id="statusServiceEndpoint"
          class="com.company.StatusServiceEndpoint">
        <constructor-arg ref="statusManager"/>
    </bean>

    <bean id="statusManager"
          class="com.company.StatusManagerImpl">
        <constructor-arg ref="statusDao"/>
    </bean>

Well that seems like a lot of spring configuration, but it eliminates a bunch of Java code. Let's break down this configuration a bit. First, the UriEndpointMapping bean. You might notice that there are some values provided from a properties file. I will be covering properties in more detail in a future blog. For now, just know spring replaces the ${data.subscriber.url} and ${request.status.url} with a valid URL value from a properties file. This defines which URLs will be routed to the specified spring integration web services inbound-gateway. The gateway will place the SOAP message onto the specified channel as a DOMSource object.

As seen above the dataGateway does not specify a marshaller and unmarshaller which is why the DOMSource object is placed on the channel. The statusChangedGateway does specify the marshaller and unmarshaller, so when it places the object on the channel, it is not a DOMSource object, but a StatusUpdate object. As you can see it is very easy to leave the input in it's native format or convert it to a Java Object. In one case the object was only going to be routed(eventually) to a JMS queue, and did not need to be unmarshalled, in the other case it required unmarshalling so that a service activator endpoint could save the object to the database. As we will see in a future blog, it would have been nice if there was a way to specify the output to the channel as a string, instead of DOMSource. This would be a great enhancement to spring integration.

The service activator is an endpoint that allows a specific method to be called, and passes the object from the incoming channel to the method. It can also place the object, either unmodified or modified, onto an outgoing channel to be further processed by the system. The rest of the configuration is very straightforward spring beans with dependency injection. This works very well if each of the web services have a unique URL, but what happens if all calls come into a single URL. That will be the topic of the next blog in this series.

Accessing External Properties Files in WebLogic

It is not only common to do, but is also a very good practice to put values that may need to change into a properties file. This creates a central location to modify values. Especially for the configuration a system based on the installation location, etc. Having the properties file inside the .war is very inconvenient when you want to change it as it causes you to recompile your code, even if it hasn't changed. I was on a project that was able to pull the property file out of the .war and put it in the JBoss AS conf directory. The application could still access it. Unfortunately the project I am on now uses WebLogic. After lots of google searching and asking a few co-workers if they new how to do it, I did find the answer.

Simply place the properties file in $WEBLOGICHOME/user_projects/domains/mydomain where mydomain is the name of he domain your app uses.

Works great and now I can change properties all day long without recompiling.

Remote Desktop & Change Password

I ran into a problem where I needed to change the password on an account that was on a server half way around the world. All I could do was Remote Desktop in to that box, and when I did Ctrl + Alt + Delete, it was changing the password on the box I was physically at, not the one in the RD session. After some searching I found this solution: Ctrl + Alt + End.

Hope this helps someone.

Java JDBC Performance vs. iBatis

While working on a recent project, I worked with a database that contained 23 billion rows in one table, and several hundred million in another table. Performance retrieving information was critical for success. I have used Spring and iBatis for several years now and only had a couple of weeks to develop a prototype, so implemented the database access layer using iBatis and spring. It is quick to implement and easy to maintain. We had several use cases we wanted to test. Some use cases returned as few as one row, some returned as many as 20 million. Unfortunately, it was taking too much time to retrieve the information for most of the use cases. One downside of using iBatis, was that I couldn't determine if the time was spent in the database query, returning the results to Java, or in creating the java objects themselves. I decided to modify the code to use straight JDBC calls so that way I could time the retrieval separately from converting to Java Objects.

I did not expect to see much performance improvement converting to straight JDBC. However, just converting to straight JDBC was a substantial performance improvement over using iBatis(30-40% faster). Unfortunately this still wasn't enough. This was a web application being deployed to WebLogic. After some research a co-worker found that the default number of rows retrieved by the WebLogic data source was 10. Queries whose results were less than 10, performed very well, but most queries returned thousands or millions of rows, so performance was very poor. We initial bumped that value to 1000, and saw another 40+% performance improvement. This in turn prompted some investigation of JDBC, and I found that the default value for JDBC against an oracle database was 10. Inside the code I increased the fetch rows value to 1000, and saw another 30% performance improvement. I concluded that more was better on the fetch rows setting, so I increased it to 50,000 and found that I now had a new problem: running out of memory. After some additional modifications to the memory available to the web application, and reducing the fetch count to 5000, I was able to get a reliable result. Essentially, a higher fetch count, meant less database connections to retrieve the data.

Using iBatis is a great way to get a lot done in a short amount of time, and for smaller databases I still prefer to use it. In addition to being easy to implement, it is also easy to maintain. However, if you have a large database, then using Straight JDBC might be a good solution. Don't forget to take fetch size into account both in the application server and in the Java JDBC code when dealing with large databases. The larger the fetch size, the fewer database connections that will be used, but the more memory that will be required. I did not investigate setting the fetch size in iBatis, so do not know if that is possible.

In the Java JDBC code setting the fetch size is done as follows:

            String query = "SELECT * FROM my_table";
            Connection conn = _dataSource.getConnection();            
            PreparedStatement st = conn.prepareStatement(query);
            st.setFetchSize(5000);
            ResultSet rs = st.executeQuery();

Some other Java improvements that helped with performance, was reducing the number of java objects being created and destroyed. This includes making any strings that are reused static member variables so they are only created once and reducing the number of java objects that are being created. If you are outputting to a file or or a message, perform those operations inline to reduce creating intermediate objects.