Bernard's Technocratic Blog

So over the weekend, our Tomcat servers were restarted without no new code deployment. For some reason exceptions were thrown in the logs indicating that the same JSP error from the previous release still existed.

The problem is that we addressed and fixed the issue on the previous release.  Furthermore, the servers were in stable condition for one week. The application is visited hundreds and thousands of times a day so the new, fixed JSP should be cached properly.

So what is the root cause?

Theory 1:

I thought that maybe when the WAR files were redeployed during the release, it was overwriting the existing WAR files. Maybe the WAR files was corrupted during this overwrite process.

Investigation: I checked the production servers and all the date stamps and file sizes on the WAR files are identical. Also, I confirmed from the release and deployment team that the WAR files and exploded WAR directories were removed before installing the new WAR files.

Theory 2:

If no new code was deployed and we are still using the existing WAR files from the last release, then there may be some weird caching issue.

So I googled it and found out that the “work” directory in the Tomcat instances need to be deleted to remove the JSP and Java caches. Why this is not automated in Tomcat, I do not know.

So here are the references that I found:

1. Old and a little relevant: http://www.coderanch.com/t/85276/Tomcat/Tomcat-weirdness-incomplete-WAR-explosion

2. “Tomcat cache – stop tomcat, delete “work” folder from tomcat install path, restart tomcat. (since this requires stopping the tomcat service, you may need an administrator’s help for this.)”
http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14292602

3. http://forums.sun.com/thread.jspa?threadID=652758

4. Even Eclipse has a build in feature to clean tomcat’s work directory: http://www.eclipse.org/webtools/releases/2.0/newandnoteworthy/server.php

5. “Have you stopped and restarted Tomcat?

Also you may need to clear the Tomcat Cache, this is located in the Tomcat installation directory in the “work” folder. Let me know if this resolves your problem”

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1256064204193+28353475&threadId=1260688

Unfortunately, the “work” directory was not cleaned during our release process.

So for now, this is what I think is the culprit to the infamous Tomcat displaying an older version of a JSP page.