MTRAT helps opensource projects find concurrent errors

Developing, testing, and debugging multithreaded programs are still very difficult; it is all too easy to create concurrent programs that appear to work, but fail when it matters most: in production, under heavy load. The most common parallel errors are data race and deadlock. These errors are typically harmful and hard to locate. Multi-Thread Run-time Analysis Tool(MTRAT) is a tool that detects and analyzes potential data race and deadlock conditions that might occur in multithreaded Java programs.

Apache FtpServer is a popular open source Java FTP server. In order to handle multiple requests at the same time, threading and concurrency are introduced in FtpServer core and its dependent library MINA, a network application framework. Let's exploit MTRAT on FtpServer to see what happens and apply MTRAT in FtpServer is easy(my operating system is Linux, so this showcase is done under Linux).

1. Download MTRAT from its webiste http://www.alphaworks.ibm.com/tech/mtrat, and configure it following the user manual. Suppose mtrat is extracted under ~/MTRAT.
   
2. The FtpServer startup script file ftpd.sh locates under bin folder, open ftpd.sh and find "$JAVACMD" -classpath "$FTPD_CLASSPATH" $MAIN_CLASS $@ in last few lines, this command is for invoking FtpServer application. Comment this line with heading "#" and add one statement to replace "$JAVACMD" with "mtrat" like this,
   ~/MTRAT/mtrat/mtrat -x java.*:sun.*:javax.*:com.*:org.eclipse.*
   :org.apache.xerces.*:org.apache.lucene.* -Dcom.ibm.mtrat.dbg.cl=false -Dcom.ibm.mtrat.novolatile=true -Dcom.ibm.mtrat.threadcache=true -Dcom.ibm.mtrat.osm=true -classpath "$FTPD_CLASSPATH" $MAIN_CLASS $@
   #"$JAVACMD" -classpath "$FTPD_CLASSPATH" $MAIN_CLASS $@
   
3. Save ftpd.sh and run it.
   

After Ftpserver launches with MTRAT, we open two new consoles and login as admin and anonymous respectively, then admin uploads a file to ftpserver while anonymous download a file from ftpserver. You will see data races are reported in Ftpserver console.

We submit two data race bugs found by MTRAT, one is in FtpServer([#FTPSERVER-122]Data races are found in FtpStatisticsImpl) while the other is in MINA([#DIRMINA-651]Data Race in org.apache.mina.core.session.AbstractIoSession), and attach the data race report generated by MTRAT.

Positive feedback is received from the Apache community, these two data race bugs are confirmed by the community leader and [#DIRMINA-651] is committed to be fixed in MINA 2.0.0-RC1 while [#FTPSERVER-122] will be fixed after component refactor.

1. "Thanks a lot for the analysis! The reason why I did not fix the race conditions already is that I'm planning to bring FtpServers statistics handling more in line with MINA.
   Anyways, I would much appreciate further analysis on the source code, especially after the statistics implementation is updated."
   -- Niklas Gustavsson
   
2. "Some of these races are in the idle time checking, which is a core functionality in MINA of course. Those we have to fix."
   -- Niklas Gustavsson
   

In addition to data race bugs found in FtpServer, Bug 45608 in Tomcat is confirmed by community too. MTRAT is proven to be practical for real-world application and useful for parallel program developers.

Multi-threaded Runtime Analysis Tool Link

http://www.alphaworks.ibm.com/tech/mtrat