Java: Generating Sequence Diagrams using jcalltracer

Github Repository : https://github.com/tuxdna/jcalltracer

This post is about getting jcalltracer to work (on ubuntu 12.04, though everything here is pretty general).

I chose it because it works by storing information on disk instead of trying to do everything in memory, so probably scales a little bit better. Also, there wasn't a lot of competition, I couldn't find that many free alternatives. It was simple to use and worked for me.

Clone the repo. Make sure you have all the prerequisites defined on the repo page.

On calling "make" I got an error, "/usr/bin/ld: cannot find -ldb" .
I had to install libdb-dev to get rid of that error.

sudo apt-get install libdb-dev

See bin/test.sh for instructions on how to use. It is self explanatory except for following..

# Run your java program, change filterList to the package name of your interest, once jvm exits, this will produce all the necessary raw information.

java -agentpath:./libjct.so=filterList=com.test,traceFile=calltrace.log -cp test-src com.test.HelloWorld

# generate sequence diagram "tree" in xml files, one per thread

bin/tree.rb

# convert sequence diagram tree to image. personally, instead of using the for loop, you should do a grep on xml files to find threads of interest and just generate images for those threads. you might have to specify more memory using -Xmx for bigger trees

for f in thread-*.xml
do
    java -jar java/dist/calltrace2seq.jar -i $f  -o output/ -f $f
done 

All the sequence diagrams would be now in output/ directory.
I found it very hard to read information from the images for big (really big) call chains. Raw tree xml file is more useful, when opened in a viewer that can display xml as a tree and you can expand/collapse elements, I simply used eclipse to view the xml.

Java SE 8 new features

Just collecting various resources containing updates for java 8. It is a major overhaul and I am so excited with all the new features those are coming especially lambdas, streams and default methods. They will help write so much better and concise code.

Here is a list from Adam L Davis's book referenced below..

- Lambda expressions and Method references

- Default Methods (Defender methods) in interfaces
With this it may seem that Abstract classes are unnecessary, but note that default methods are introduced for interface evolution and Interfaces are still more restricted than Abstract classes due to following.
-- Abstract classes can define constructors while Interfaces can't.
-- Fields inside Interface are always considered public and final, so Interfaces can't have a mutable state.


- Static methods in interfaces

- A new Stream API.

- Optional

- A new Date/Time API.

- Nashorn, the new JavaScript engine

- Removal of the Permanent Generation

References:

What's New in Java 8 by Adam L Davis

Oracle's what's new in Java 8 page

Java 8 Docs

Good Unit Tests

Found a good thread on stackoverflow regarding the good unit tests : http://stackoverflow.com/questions/61400/what-makes-a-good-unit-test

 Pasting the most important ans from above thread here..
--------------------------------------------------
Good Tests should be A TRIP (The acronymn isn't sticky enough - I have a printout of the cheatsheet in the book that I had to pull out to make sure I got this right..)

• Automatic : Invoking of tests as well as checking results for PASS/FAIL should be automatic
• Thorough: Coverage; Although bugs tend to cluster around certain regions in the code, ensure that you test all key paths and scenarios.. Use tools if you must to know untested regions
• Repeatable: Tests should produce the same results each time.. every time. Tests should not rely on uncontrollable params.
• Independent: Very important.
  • Tests should test only one thing at a time. Multiple assertions are okay as long as they are all testing one feature/behavior. When a test fails, it should pinpoint the location of the problem.
  • Tests should not rely on each other - Isolated. No assumptions about order of test execution. Ensure 'clean slate' before each test by using setup/teardown appropriately
• Professional: In the long run you'll have as much test code as production (if not more), therefore follow the same standard of good-design for your test code. Well factored methods-classes with intention-revealing names, No duplication, tests with good names, etc.
• Good tests also run Fast. any test that takes over half a second to run.. needs to be worked upon. The longer the test suite takes for a run.. the less frequently it will be run. The more changes the dev will try to sneak between runs.. if anything breaks.. it will take longer to figure out which change was the culprit.

• Readable : This can be considered part of Professional - however it can't be stressed enough. An acid test would be to find someone who isn't part of your team and asking him/her to figure out the behavior under test within a couple of minutes. Tests need to be maintained just like production code - so make it easy to read even if it takes more effort. Tests should be symmetric (follow a pattern) and concise (test one behavior at a time). Use a consistent naming convention (e.g. the TestDox style). Avoid cluttering the test with "incidental details".. become a minimalist.

Apart from these, most of the others are guidelines that cut down on low-benefit work: e.g. 'Don't test code that you don't own' (e.g. third-party DLLs). Don't go about testing getters and setters. Keep an eye on cost-to-benefit ratio or defect probability.
-----------------------------------------------

Some of my own from experience(not strict rules that I follow but just guidelines)..

- Whenever writing tests, keep in mind, how much the test code changes if you change the implementation(keeping the interface same) of your method under test. In short, try to test specification as much possible.

- Recently, to abstract away a key-value store, I created an interface, KeyValueDb. In addition to the real key-value store based implementation I wrote another one which is backed by memory. All the code that uses KeyValueDb, I used memory-backed implementation in the unit tests and surprisingly tests look much cleaner than using mock(KeyValueDb). However, mocks are required in some places to create scenarios hard to create otherwise(e.g. socket timeout).

Also, though I don't do T.D.D.[neither Design nor Development], but writing unit tests while/after I am done coding something gives me confidence that it works. It sure helps catch bugs and sometimes results in good method level refactorings .

Not tired of reading yet? Here is another intersting post... http://mike-bland.com/2012/07/10/test-mercenaries.html .

ad hoc jvm monitoring and debugging

Many a times we notice some issue in production environments and quickly need to investigate the details with ssh connection to a terminal. jdk comes packaged with some very useful tools to monitor/debug jvm which can come in handy..

jps - lists the vm-id's of instrumented jvm processes running on the machine

jinfo - gives you a lot of detail about the configuration a jvm process was started with(which, otherwise, is scattered around innumerable number of configuration files and environment variables)

jstat - a jvm monitoring tool, very quick to setup. it can basically monitor the heap, gc frequencies and time elasped doing gc. It can be setup with one command e.g. "jstat -gcold -t vm-id 10s"

jstack - this lets you print the stack trace of all the threads. it can automatically find out the deadlocks. Also, you can find high contention locks by taking multiple thread dumps over a small period of time and see which locks are being waited upon most frequently. Use "jstack -F -m vm-id". Use additional "-l" option to print lock information (it will be slow though).

jmap - basically the heap analyzer. among other things, you can use it to dump the contents of heap to a file(e.g. jmap -dump:format=b,file=mydump.hprof vm-id). you can use jhat to explore this file using a browser or use eclipse-mat that gives better ui and functionality.

hprof -  standard java cpu/memory profiler bundled with jdk. this is not really ad-hoc as you would have to add it to jvm options at start up instead of just attaching at will. output can be processed via jhat and other fancy tools such as yourkit.
java -agentlib:hprof=help
java -agentlib:hprof=heap=sites,cpu=samples,depth=10,monitor=y,thread=y,doe=y,format=b,file=/home/himanshu/profile_dump.hprof

http://www.javaworld.com/article/2075884/core-java/diagnose-common-runtime-problems-with-hprof.html
http://docs.oracle.com/javase/8/docs/technotes/samples/hprof.html

Note that, when the jvm process is started by a different user than the one you are logged in with, your logged-in user might not have permissions to attach to the jvm process and you may need to use sudo with all of the above commands.

Btw, these tools are not limited to jvm processes running locally but can be used with remove jvm processes as well using rmi. In this case you could use graphical clients JConsole and JVisualVM also.

A bit orthogonal to jvm monitoring, but following are some noted jvm startup options that are very helpful when things go wrong.

-XX:ErrorFile=/path/to/hs_err_pid.log
If an error occurs, save the error data to given file.

-XX:-HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/path/to/java_pid.hprof
Dump the heap to given file in case of out of memory error.

-XX:-PrintGCDetails
-Xloggc:/path/to/gclog.log
Prints useful information about gc in given file. you can use gcviewer to analyze this file.

References:
http://docs.oracle.com/javase/8/docs/technotes/tools/
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html

Dynamic Proxies in Java

Java Proxy class in reflection package lets you dynamically create a proxy class(and its instances) that implements interfaces of your choice. Proxy instance contains a InvocationHandler supplied by you. Any method calls to the proxy instance call the handle method of passed invocation handler where you can determine the behavior that you want on the call.
So, for example, you can very easily wrap around implementation of an interface and do some magic before/after the method invocation(and much more of course). Using this you can get lightweight "aop" like functionality for method interception.

A much more informative article on the same topic is http://www.ibm.com/developerworks/java/library/j-jtp08305/index.html

Java SE 7 new features

JDK7 has got many new features and folks have done a great job of writing good articles about all of them, so I wouldn't repeat any of that.
For my quick access, I will keep a reference to many such articles and a quick list of interesting features.

- Type inferencing for generic instance creation
- Strings in switch statements
- try-with-resources and using exception suppression instead of masking it
- Catching multiple exceptions in single catch block
- NIO 2.0 changes
- Fork-Join paradigm

References:
Good article on try-with-resources and exception suppression
http://www.oracle.com/technetwork/articles/java/trywithresources-401775.html

Fork-Join article
http://www.cs.washington.edu/homes/djg/teachingMaterials/spac/grossmanSPAC_forkJoinFramework.html
 http://stackoverflow.com/questions/7926864/how-is-the-fork-join-framework-better-than-a-thread-pool

Misc
http://radar.oreilly.com/2011/09/java7-features.html

jvmstat code demo

In last post, I showed how you can connect to a locally running jvm's jmx agent using java attach api and get various monitoring statistics from the in built jmx beans. In addition to jmx beans, you can gather similar information from jvm using the sun.jvmstat.monitor pacakge.

import sun.jvmstat.monitor.Monitor;
import sun.jvmstat.monitor.MonitoredHost;
import sun.jvmstat.monitor.MonitoredVm;
import sun.jvmstat.monitor.VmIdentifier;

/**
 * This example demonstrates the usage of sun jvmstat package to connect to a jvm
 * running locally(given the vm id obtained using jps command) and print out the
 * count of live threads.
 * 
 * You have to add $(jdk-home)/lib/tools.jar in your class path to compile/run
 * this code.
 *
 * @author Himanshu Gupta
 */
public class JvmStatDemo {

    public static void main(String[] args) throws Exception {
        String id = args[0];
        VmIdentifier vmId = new VmIdentifier(id);
        MonitoredHost monitoredHost = MonitoredHost.getMonitoredHost(vmId);
        MonitoredVm monitoredVm = monitoredHost.getMonitoredVm(vmId, 5);
        Monitor m = monitoredVm.findByName("java.threads.live");
        System.out.println("Number of live threads = " + m.getValue());
        
        //You can print whole list of monitored stuff
        /* List logged = monitoredVm.findByPattern(".*");
        for(Iterator i = logged.iterator(); i.hasNext(); ) {
            m = i.next();
            System.out.println(m.getName() + " : " + m.getValue());
        } */
    }
}

Jstat uses above mechanism to give locally running jvm monitoring information.

Java Attach Api demo

This is a quick demo of Java Attach Api . Jvm comes instrumented with many very useful monitoring jmx beans. In this example, given the vm id(that can be quickly obtained using jps command, in fact attach api can be used to list all the jvm processes running too), We get a proxy instance of ThreadMXBean and print the count of live threads.

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;

import javax.management.MBeanServerConnection;
import javax.management.ObjectInstance;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import com.sun.tools.attach.VirtualMachine;

/**
 * This example demonstrates the usage of java attach api to connect to a jvm
 * running locally, gets handle to ThreadMXBean proxy and prints the count
 * of live threads.
 * You have to add $(jdk-home)/lib/tools.jar in your class path to compile/run
 * this code.
 *
 * @author Himanshu Gupta
 */
public class JmxWithAttachApiDemo {

    public static void main(String[] args) throws Exception {
        String vmid = args[0];
        VirtualMachine vm = null;
        try {
            vm = VirtualMachine.attach(vmid);
        } catch(Exception e) {
            System.out.println("Failed: could not find/attach vm with id: " + vmid);
            e.printStackTrace();
            System.exit(1);
        }
        
        JMXConnector jmxConn = null;
        try {
            String connectorAddress = vm.getAgentProperties()
                    .getProperty("com.sun.management.jmxremote.localConnectorAddress");
            if(connectorAddress == null) {
                System.out.println("Failed: jmx agent does not seem to be running.");
                System.exit(2);
            }
            
            JMXServiceURL url = new JMXServiceURL(connectorAddress);
            jmxConn = JMXConnectorFactory.connect(url);
            
            MBeanServerConnection mbs = jmxConn.getMBeanServerConnection();
            if(mbs == null) {
                System.out.println("Failed: could not get mbean server connection.");
                System.exit(3);
            }
    
            ObjectName on = new ObjectName("java.lang:type=Threading");
            ObjectInstance oi = mbs.getObjectInstance(on);
            ThreadMXBean bean = ManagementFactory.newPlatformMXBeanProxy(mbs,
                    "java.lang:type=Threading", ThreadMXBean.class);
            if(bean == null) {
                System.out.println("Failed: could not get threading mbean.");
                System.exit(4);
            }
            
            System.out.println("Number of live threads = " + bean.getThreadCount());
            System.exit(0);
        } finally {
            if(vm != null)
                vm.detach();
            
            if(jmxConn != null)
                jmxConn.close();
        }
    }
}

This can be used to build jvm monitoring tools and indeed,   JConsole and JVisualVM work this way.



References:
http://docs.oracle.com/javase/7/docs/technotes/tools/
http://docs.oracle.com/javase/7/docs/technotes/guides/management/toc.html

easy deadlocking in java

Traced down a stupid dead lock in an application today(thank god, jstack exists), I never realized it was so easy to produce a dead lock with *one* thread.

Look at the following code...

public class Main {

    public static void main(String[] args) throws Exception {
        
        ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
        ReadLock rlock = lock.readLock();
        WriteLock wlock = lock.writeLock();
        
        try {
            System.out.println("Aquiring read lock...");
            rlock.lock();
            System.out.println("Aquiring write lock...");
            //gets blocked here
            wlock.lock();
            System.out.println("Got write lock as well");
        } finally {
            wlock.unlock();
            rlock.unlock();
        }
        System.out.println("Finished.");
    }
}

Above code gets blocked at wlock.lock() as read lock can not be upgraded to write lock.

Whats worse is that other threads will now block even to acquire a read lock because main thread is waiting to acquire a write lock. Essentially, with all likelihood, your whole application will stop working ;).

 Here is the code demonstrating that other thread will block even for a read lock when nobody really has a write lock(and this behavior is implemented to save the thread, waiting to acquire write lock, from starving).

public class Main {

    public static void main(String[] args) throws Exception {
        
        ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
        ReadLock rlock = lock.readLock();
        WriteLock wlock = lock.writeLock();
        
        try {
            System.out.println("Aquiring read lock...");
            rlock.lock();
            
            tryAcquiringReadLockInAnotherThread(rlock);
            System.out.println("Aquiring write lock...");

            //gets blocked here
            wlock.lock();
            System.out.println("Got write lock as well");
        } finally {
            wlock.unlock();
            rlock.unlock();
        }
        System.out.println("Finished.");
    }
    
    private static void tryAcquiringReadLockInAnotherThread(final ReadLock rlock) {
        Thread t = new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    Thread.sleep(1000);
                    System.out.println("Trying to aquire read lock in other thread...");
                    //gets blocked here
                    rlock.lock();
                    System.out.println("Acquired read lock in other thread.");
                } catch(InterruptedException ex) {
                    //do not do this in real program :)
                    ex.printStackTrace();
                }finally {
                    rlock.unlock();
                }
            }
        });
        t.start();
    }
}

I tried it on following java version..

$ java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Ref: ReentrantReadWriteLock

mvn helpers

recently needed to setup/debug/refactor a complex multi-module maven project.. some maven stuff, learned the hard way....

Generate eclipse setup files in a maven project...
mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs

Default eclipse may point M2_REPO to ~/.m2 . If you want to change that, you have to do so in Windows->Preferences->Maven->User Settings .

Skip tests while building the project..
-Dmaven.test.skip  skips both test compilation and execution
-DskipTests=true    skips test execution only


Run tests from named test class only..
-Dtest=<fully-qualified-test-class-name>Executing a class containing main(String[] args) method..
mvn exec:java -Dexec.mainClass=<fully-qualified-main-class-name>

Build selected modules of a parent pom..
mvn package -pl mod1, mod2 [-am]
-pl : list of modules
-am : also make the dependencies

Build starting from a given module
mvn package -rf :mod3

Downloading dependencies of the project
mvn dependency:copy-dependencies -DexcludeScope=provided


(More at http://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html )
 
Getting description of goals supported by a plugin
mvn help:describe -Dplugin=<name> (if plugin group has been added to pluginGroups, look in settings reference below)
or
mvn help:describe -DgroupId=<groupId> -DartifactId=<artifactId>
.. add -Dfull to get more detailed information on goals.

Check out other goals in help plugin itself
mvn help:describe -Dplugin=help -Dfull

some important ones are...

mvn help:effective-settings
mvn help:active-profiles
mvn help:effective-pom

Finding dependencies of your project
mvn dependency:analyze
mvn dependency:tree
dependency graph: http://mvnplugins.fusesource.org/maven/1.8/maven-graph-plugin/


You can also remotely debug the tests run through maven by using
-Dmaven.surefire.debug
 flag. Optionally you can specify all the jvm debugging related parameters using
-Dmaven.surefire.debug="

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE

" (Reference)

You can run a specific testcase with -Dtest flag as described on Ref . Tests can altogether be skipped using -DskipTests flag.
For testng, you can annotate a test like @Test(groups = {"xyz"}) and then just run these tests by using flag -Dgroups=xyz (remember to escape '=' in shell by using '\=')

try using "mvn -U" to force remote repository snapshots update when mvn is failing to find an artifact.

packaging all the dependencies into jar: 
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
        <executions>
            <execution>
                <id>make-my-jar-with-dependencies</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

Downloading all dependencies--
copies all dependencies to target/dependency, See plugin doc for more options to refine what is copied.
mvn dependency:copy-dependencie [-DexcludeTransitive -DoutputDirectory=<some_dir>]


Resources:
Pom Reference
Settings Reference

Books:
http://maven.apache.org/articles.html
http://www.sonatype.com/books/mvnref-book/reference/

embedding activemq in jboss

Today, I worked on a prototype web application that utilized JMS. We decided to use ActiveMQ for this. Here are the things I had to do to get it running embedded inside Jboss.

First, I followed the process described in the doc "Integrating ActiveMQ with Jboss".

By default ActiveMQ uses KahaDB for persistence, but I wanted to use our oracle db infrastructure for this. A bit of googling got me to Introduction to ActiveMQ persistence and that led to JDBC data source configuration . By now I got the ActiveMQ successfully embedded in Jboss and running.

Next, Instead of giving all the oracle db info in the broker-config.xml, I wanted to get it from Jboss managed jdbc datasource(which is being used by other applications also). So, I have already got mydatasource-ds.xml deployed inside deploy/ folder of jboss which looks like following

<datasources>
    <local-tx-datasource>
        <jndi-name>jdbc/mydatasource-ds</jndi-name>
 <connection-url>jdbc:oracle:thin:oracle_user/passwordu@host:1521/sid</connection-url>
        <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
        <min-pool-size>1</min-pool-size>
        <max-pool-size>20</max-pool-size>
        <metadata>
            <type-mapping>Oracle10g</type-mapping>
        </metadata>
    </local-tx-datasource>
</datasources>

In order to use above datasource, Here is what I had to put inside activemq-ra.rar/broker-config.xml file.

<beans ...>

  <!--  shutdown hook is disabled as RAR classloader may be gone at shutdown -->
  <broker xmlns="http://activemq.apache.org/schema/core" useJmx="true" useShutdownHook="false" brokerName="myactivemq.broker">
    ...
    ...
    <persistenceAdapter>
       <!-- kahaDB directory="activemq-data/kahadb"/ -->
       <!--kahaDB directory="${jboss.server.data.dir}/activemq"/ -->
       <jdbcPersistenceAdapter
           dataDirectory="${jboss.server.data.dir}/activemq"
           dataSource="#oracle-ds"/>
    </persistenceAdapter>

   ...
  </broker>

  <bean id="oracle-ds" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:jdbc/mydatasource-ds"></property>
  </bean> 
</beans>

Well, above did not work. What happened was that Jboss was trying to start ActiveMQ before deploying mydatasource-ds.xml which resulted in JNDI NameNotFoundException(as java:/jdbc/mydatasource-ds is not available until mydatasource-ds.xml gets deployed). So, in ActiveMQ configuration, I had to declare dependency on mydatasource-ds(so that it gets deployed first). To do it, I created a file named jboss-dependency.xml inside activemq-ra.rar/META-INF/ and pasted following on it.

<dependency xmlns="urn:jboss:dependency:1.0">
  <item whenRequired="Real">jboss.jca:service=DataSourceBinding,name=jdbc/mydatasource-ds</item>
</dependency>

Now, We are good :).

BTW, above is by no means an exhaustive list because as we go, we will make a lot of changes for the actual setup but above should get a new person started with less amount of trouble than I had.

concurrency concepts in databases and hibernate

I've read about this multiple times but often forget the details as it is not often that I need to work on it in my applications. So, here I am taking quick notes for faster recap of it later.

First thing to know is what are the issues that can happen by multiple concurrent transactions if no preventive measures are taken.

lost update : I have found different meanings to this term. The hibernate book describes this as follow. One transaction commits its modifications and another one rolls back resulting undoing of the updates done by the other transaction.
However at many other places lost-update has the meaning that of second-lost-update described later. And, ANSI SQL-92 standard does not seem to talk about it.

dirty read : One transaction fetches a record and modifies it. Another transaction fetches the same record and gets the modifications made by first transaction even if they are not committed.

unrepeatable read : One transaction reads some record, another transaction commits some modifications to same record, first transaction reads the same record again and finds it different from what was read previously.
One special case of unrepeatable read is the second lost updates problem. Let say, One transaction reads some record and modifies it, another transaction reads same record and modifies it. First one commits its modifications and then the second commits and modifications made by first transaction are lost.

phantom read : One transaction reads a list of records by some filter criteria. Another transaction starts, inserts some records that fulfill the filter criteria used by first transaction and commits. First transaction reads same list again by same filter criteria but ends up getting a different set of records this time.

Now, to overcome above issues ANSI standard defines following transaction isolation levels.

read uncommitted : this permits all 3.. dirty read, unrepeatable read and phantom reads.

read committed : this does not permit dirty reads but allows unrepeatable as well as phantom reads.

repeatable read : this does not permit dirty and unrepeatable reads but allows phantom reads.

serializable : this is the strictest transaction isolation which simply does not seem to allow concurrent transactions. this again is usually not used in typical applications due to being too restrictive and badly performing. This does not allow any of dirty read, unrepeatable read or phantom reads.

exact implementation of above isolation levels varies significantly among the vendors. One has to consult the docs of particular db to understand the impact of each isolation level to performance and scalability.

In Hibernate, by default, every JDBC connection to a database is in the default isolation level of the DBMS, usually read-committed or repeatable-read. You can change this by explicitly setting hibernate.connection.isolation property. Then hibernate will set the mentioned isolation level to each new JDBC connection(note that it will not change the default isolation level of your db).

Optimistic concurrency control:
An optimistic approach assumes that everything will be OK and any conflicts are detected only in the end when data is written/flushed to the db.
Multi-user applications usually default to optimistic concurrency control and database connections with a read-committed isolation level.
Let say the isolation level is set to read-committed. Transaction A and B start at the same time and both read and modify the same record(they can not see each other's changes as dirty read is not permitted in read-committed isolation level). Transaction A commits and then B does. Then one of the following 3 things can happen.

last commit wins : both transactions commit successfully and B overrides any modifications done by A and no error raised.

first commit wins : when second transaction(B) is committed, conflict is detected and error is raised.

merge conflicting updates : conflicts are detected and one interactive dialogue helps resolving those conflicts

With hibernate, you get last-commit-wins by default. You can enable first-commit-wins strategy by enabling optimistic concurrency control. This needs versioning enabled for entities .


In the end, if you need more fine grained control on locking then you can use variety of "SELECT ... FOR UPDATE ..." statements by using LockMode.UPGRADE et al. This is called explicit pessimistic locking.


Reference:
Chapter-9: Transactions and Concurrency in the book "Java Persistence with Hibernate".
ANSI SQL-92 Standard Spec

reliability attributes

Often times in the discussion of distributed systems and scale out architecture, we demand system being reliable. One of the application, I was part of designing/developing the scale out architecture for it primarily to fulfill the business demand that we should be able to add more hardware to support more load. Now, recently I watched this video from cloudera's free basic hadoop training and it talks about "reliability attributes". I realized that we need same reliability attributes in scale out architecture also and often talk about them( implicitly) in my team. And, this list makes them explicit(coming straight from the linked video)...

Partial Failure: System should be able to support partial failures. That is, if x out of n nodes(where x < n) in the system go down then only system's performance(e.g. throughput) should gracefully go down in the proportion of x/n instead of it being go down completely and not doing any work(or not serving any requests.

Fault Tolerance: This is more related to background jobs, so map-reduce in particular. If one of the nodes go down then its work must be picked up by some other functional unit. This is also sometimes solved by having redundancy in the system. For example, within one cluster, we have multiple app servers serving same set of requests and a load balancer to manage them. In case, one of the server goes down, load balancer detects it and stops sending any requests to it.

Individual Recoverability: Nodes that fail and restart should be able rejoin the group(or cluster) without needing a full restart.

Consistency: Internal failure should not cause externally visible issues.

Scalability: If we add more nodes, we should be able to handle more load in proportion of the new nodes added.

reasons I will not use flex for

This is not an anti-flex post but the list of issues I *personally* dealt with and would avoid using flex for. Of course there are work arounds for all of these but I would expect them to be available out of the box.

1. Flex does not send "cookies" header while "file upload" using FileReference.upload() in Firefox.

2. If you make a http request from flex app, you only get success/failure and not HTTP status code.

3. Flex application compiles into "swf" files which are rather large for a web application and for the people using your website with lower bandwidths it becomes terribly slow. (it may sometimes need to download the flex framework files also if the person has never visited a site having flex content before)

some more demotivator types..

4. It simply can not be debugged on the fly in the browser. For javascript, now a days you can do all the debugging on the browser itself and it becomes so easy.

5. FlexBuilder commertial license costs money :).

6. Personally, as a developer, I would rather want to learn/master javascript than actionscript.

7. World is pushing HTML[5]/Javascript/Css[3] hard. Javascript browser implementations, UI libraries/widgets are only going to get better and faster.

We chose flex over html/javascript because it seemed that it would be easier/quicker to create cool UI(RIA) with flex than html/javascript. But, now we believe same and more cool UI are possible and easily doable(with so many options, jqueryUI, GWT etc) with html/js also.

sql "IN" clause gotcha

Though standard sql puts no limit on the number of entries you can put inside an IN clause, but various vendors typically put a limit. Oracle puts a limit of 1000 to it. This caused a bug in our app as we dynamically generate a sql query where number of entries went beyond 1000 today.

You can put any one of the resolution stated below(in decreasing order of preference)

1. Change your logic so that number of entries never go beyond 1000(it may be a good practice in general to follow).
2. See if you can use "BETWEEN" clause for your particular case.

Instead of using "Select blah from tablexyz WHERE col IN (c1,c2,....c50000)"
3. Use "Select blah from tablexyz WHERE col IN (select id from ctable)" and the subquery "select id from ctable" returns resultset containing c1,c2,...c50000.
4. Use "Select blah from tablexyz WHERE col IN (c1,c2,...c1000) OR col IN (c1001,c1002,...,c2000) OR ... OR col IN (c49001,c49002,...,c50000)".
5. Use multiple sql queries each with one IN clause containing 1000 entries.

Much of above came from this thread.

web application logging dilemma

By exploring languages like lisp, scheme, scala etc I've come to disliking "verbose code" and I try to write less code as much possible(I still follow the KISS principle though, so no clever tricks unless really necessary). But industry(and day to day) reality is Java, and "verbosity" manifests itself in java no matter what you do.
Typically, Writing a debug log in a java application takes *at least* 2 lines..

if(log.isDebugEnabled())
   log.debug("your error message" + usually_a_concatenation)

As a result, My less verbose code instinct makes me write fewer and fewer debug logs and lets me write only the necessary error logs because number of lines just explode when you write too many logs and doesn't look good. And also, more logs hamper performance.

I am writing this post because last night I spent 2 hours figuring out an issue in staging servers. It took so much time because there were not enough log statements(so I had to do a lot of looking into the code, guess work and trials to figure out where the problem might be) and in an enterprise setup it becomes a very slow process to try stuff at stage/production servers.

The lesson learned is that, there should be enough(infact alot of) logs for the whole initialization process. So that you don't have to waste time guessing. Since they will get printed only on server startup so they wouldn't hamper the performance of the over all system. And enough debug logs per request, so that issues can be tracked down easily by printing debug logs. Though it will increase the number of lines, will not look good but it is probably worth it :(.

mistakes made in DDL script and lessons learned

Last year, I got the opportunity to create a java web application from scratch. Here, I want to talk about what I should have done differently when first DDL script for the app was created.

I'm correcting all these things now, but this is the technical debt that could have been avoided.

note: some of the jargons might be oracle specific

1. Create separate tablespaces for table, index and lobs.
This is a standard practice and helps DBAs in a lot of ways with management of db. The one I created had USERS as its default tablespace and none of CREATE TABLE, INDEX etc statements said anything about tablespace and everything resulted in USERS tablespace.

2. Give upfront thought to version upgradation from db perspective.
Once your application goes to production, and db gets loaded with a lot of data. It becomes rather difficult to upgrade schema from previous version of application to newer one. Its very important to have your migration strategy thought through and in place from the beginning itself.

3. Do not ignore "ON DELETE" clause when creating foreign key constraint.
In Oracle, you got two options here - ON DELETE SET NULL, ON DELETE CASCADE
you should be choosing one over the other based on how you want your data to evolve for the table.

4. Keep two users to manage your db.
One is the schema owner who runs all the DDLs and another one the schema user who has just enough privileges to run the application. Application should use the "schema user" user only to save application cdoe from getting any unwanted access to the db. Typically, schema user will need following privileges.

- CREATE SESSION
- CREATE SYNONYM
- SELECT/UPDATE/INSERT/DELETE privilege on all tables
- SELECT privilege on all SEQUENCEs

Performance Tuning

I'm scribbling here my recent(and ongoing) experience with performance tuning a java web application. Like any other article on optimization, I'll start with the quote, "Premature optimization is the root of all evil.". My take is that don't *optimize* prematurely, but do not make it an excuse for *not measuring* the performance of your system. It is good to know how your system performs at all times.

Anyway, here is the common way(was reinforced after attending a talk on performance tuning in devcamp-2010) that everyone knows about but I'm just putting here anyway...

1. Pick conservative hardware and install your application on it.

2. Create a large sample(but realistic) dataset. It might be good to just take a dump from production and load your system with it.

3. Put your system to extreme load. In general the load will be very general doing all the functions on your system. But, in the special case where you're just exploring one feature's performance/GC-activity then put the load on that feature only. Use any of the load generation tools... Apache Bench(non-gui) and Apache Jmeter(gui as well as non-gui modes) are two free ones.

4. Measure, Measure and Measure.
This gets tricky, I've tried to use hprof for memory dumps, Runtime.freeMemory and Runtime.maxMemory methods for memory consumption analysis but ended up using a profiler(a commertial one that is standard in our org) and it really helps.

5. Make your change if any and then measure again and keep repeating.

6. Automate, as much possible, the whole process for faster feedback loops.

So far we've found following major boosters..

1. Putting indexes, we always were very careful while putting indexes as they become overhead for write/update heavy tables, but later found there are many tables where indexes are helping alot more than we imagined.

2. We noticed that alot of garbage was generated and most of it was char[] data resulting from a lot of Strings. And finally figured out that the culprit was heavy amount of logs *printed*. However, Note that log statements in the code are not that expensive but printing them is. So, for prod/perf environment set print log level to ERROR and be very careful while putting log statements in app(I mean being judicious about what level you print the log in.. DEBUG/INFO/WARN/ERROR/FATAL), anything in log.error should really be an error.

3. Guarding log.debug statements with log.isDebugEnabled.

4. Caching with LRU(policy may very well depend on particular use case)

[Update:] We realized that our background jobs are very memory/cpu hungry, so we created a separate job node to run just the background jobs. It is improving the performance of serving synchronous client requests tremendously. However, we are still working on to make background jobs more efficient in terms of memory/cpu usage.

[Update:10-12-10] We realized that we usually used bind variables in sql/hql queries when wrote them by hand but when generated dynamically, we forgot(its easy) to use bind variables. Also, we never used bind variables wherever we had IN clauses. Since we all know that it is more efficient to use bind variables to reduce parsing times in db(they get more cache hits on already parsed sqls), So we are paying this technical debt and correcting all of this now. While we're doing it, it seems it would have been a simpler exercise if all the hql and sql statements were written in one place then it would have been easier to go through all of them as right now they are all scattered in different places in the big project code-base

[Update:14-12-10] We are resorting to use bulk manipulation queries and they are giving significant improvements. To take an extreme example, it is much efficient to directly execute query, "UPDATE TABLE t SET c = 'blah'", than iterating over every data object, updating it and saving it(unfortunately, ORMs give us the habit of thinking always in terms of objects and we forget to use these simple optimizations).
Also, I find the contents on this stackoverflow thread very informative and It seems I've also made some of the mistakes mentioned there :).

[Update:10-03-11] Never leave memory requirements of an operation unbounded. If your memory requirement increases in proportion to some dataset size and you believe that dataset size is upper bounded then place some mechanism so that if the dataset size goes beyond your assumption then somehow you get the notification. In my experience, usually my assumptions were either wrong or with time the load grew and assumption became wrong.. so it is much better if you plan your code so that dataset size is also bounded by design rather than by belief :). For example if you're fetching a list of records from database, probably there will be good enough reasons to use pagination parameters.

Issues faced with Mule-2.2.1

Mule is one of the open-source(commercial support available) ESB. These are some of the showstoppers I faced while I used mule in one of my projects. (Things may get resolved in future versions though).

1. It does not support multiple headers. So, what it means is that, If you're doing some kind of login and your server sends multiple Set-Cookie header to mule in response then mule will ignore all but last Set-Cookie header. To add to the trouble, you'll read RFC-2109 and believe that concatenating all the cookies(using comma as separator) inside single Set-Cookie header will resolve it but it'll not as browsers don't honor the spec in this regard.



2. You can't set virtual host/port on a http request made from mule. So, if you want to use mule as a proxy to transparently pass the host header to the end server, well it will not.



3. Do not use MuleMessage.getPayloadAsString() in case your payload is Input-Stream, as it will read the whole stream ignoring Content-Length "property" completely and that will result in issues if your response was a http message with large binary content. Content-Length might also get messed up in the end response.



4. It is completely unintuitive, you will end up writing too much code inside xml configuration.

globalizing a web app

I'm involved in globalizing a web-app, which was not designed keeping g11n in mind. This material might be trivial but I'm noting down here things that are coming up(so that its quicker/easier the next time).

We realise, we need to store 4 user preferences(at least) to make decisions on what to do/show

Country
Locale
Currency
Time-Zone

We're putting following things in, stuff is chosen based on logged in user's preferences.

For Client Side:
Resource Bundles per country-locale
CSS per country-locale
Display Images per country-locale
Client Side validations per country-locale

For Server Side:
App properties per country-locale
A Validation framework where we can write different validation rules for different country-locale pair
Each date is stored as long + timezone string
Oracle DB instance needs to support UTF-8 encoding and VARCHAR2 with char semantics