The incredible zip -u

09 Jul 2014 - Marcelo Costa

Quickly re-packaging jar/libs to debug Java apps

The customer called… according to him, a functionality that was working fine, mysteriously stopped working and it is up to you to investigate the problem. Just a reminder that you have those 17 items in your backlog that were supposed to be finished yesterday and this new incident is high priority, the phone won’t stop ringing, several business users are calling, the manager is coming towards your desk, the walls are closing in… you just found the component and… look at that? The company was outsourcing the project so It was developed by some other team back in ancient times. That’s just great! In summary: Just another happy day in the IT world.

Ok, so here’s one of those posts where I share a walk-through on how to troubleshoot a problem with a Java Web Application. Let us assume that you already mapped the issue on the client-side, opened your browser’s Web Debugger (e.g., Firefox’s Firebug or Chrome’s Dev tools), read the Javascript code and found out that something in the back-end is messed up and it is not producing the expected result.

When you move on to the server-side, bear in mind that, in order to understand what the application is doing, you will have to READ CODE. Unfortunately, it is common to find situations where comprehensive up-to-date documentation is not available.

Let’s say that a class called “ChaoticUtils” shows up in some stack trace in the Application Server’s logs, how can we find the class and investigate further? Assuming that you don’t have access to the source, or, even worse, you can’t trust the release management process! In such cases it’s best to just get the JAR and decompile the class (you will be 100% sure you are looking at the exact same code that is being executed there). However, as you know, in the Java world it is common to have lots and lots of JAR packages that can be used for extensions/dependencies. In order to find the JAR package of a specific class, you can use this tool: “Whereis.jar“. Here’s how to use it:

$ java -jar whereis.jar CaothicUtils /opt/company/webapps/

Here’s the output generated by this tool:

/opt/company/webapps/ext_ear/APP-INF/lib::ext-links.jar::com/company/xxx/utils/CaothicUtils.class

Now let’s decompile this class to see what is going on. I recommend this tool called “JD Decompiler“, just load the .jar package into the program and it will decompile all the classes. Here’s a screenshot of its interface:

In case the root cause can’t be identified straight away just by looking at the code, you can introduce additional logging lines, e.g., System.out.println will print messages to the standard output (stdout) of the Application Server. So the objective would be to change the class or a JSP that is stored within a WAR or a JAR package to add these additional log messages, and, to speed things up, we are going to do that without having to unpackage, change the files and recreate the package.

So, without further ado, I introduce you to the incredible ‘zip -u‘!

For the ones that have some experience with Linux, the zip command is something very common, but I would like to focus on a specific parameter of this command: -u.

-u Replace (update) an existing entry in the zip archive only if it has been modified more recently than the version already in the zip archive. For example:

zip -u stuff *

Imagine that, in order to debug a JSP or a class, we would have to insert some lines of code to see if the variables were being populated accordingly, we certainly don’t want to unpackage every .war or .jar to insert a modified JSP or class during the troubleshooting.

For situations like that, you can use “zip -u”.

Of course… using the decompiled code to perform Remote Debugging is also an option, you can just set some breakpoints in your IDE and visualize the values of each variable through the DEBUG interface, here are the JVM arguments that you can use in order to do that:

-Djava.compiler\=NONE -Xdebug -Xnoagent -Xrunjdwp\:transport\=dt_socket,server\=y,address\=6065,suspend\=n

Anyway, in order to update, let’s say, a JSP inside a .war package we need to create the same folder structure outside the war and place a copy of the JSP inside this structure, for example:

chaotic.war
|– scripts
|– common
.  |– jsps
.    |– links
.      |– example.jsp
|– WEB-INF

If you are in the same directory as the ‘chaotic.war’ file, use the command “mkdir -p jsps/links” to create the folder structure and then copy the JSP to the ‘links’ folder. After you modify the JSP using your favorite editor (e.g., Emacs or vi) you just need to run the following command:

$ zip -u chaotic.war jsps/links/example.jsp

You should see something like this:

updating: jsps/links/example.jsp (deflated 66%)

** Bear in mind that it is necessary to redeploy the .ear package if you are updating a .jar inside ‘APP-INF/lib’ or the .war if you are updating a JSP or a .jar package, this is done through the Application Server, that’s how the changes take effect.

You can follow the same approach for classes inside a .jar package. If there is no Logger configured in the class and you are adding the good ol’ System.out.println() all over the code, you can discover what is the standard output (stdout) of the messages that are being generated by this class by following this approach, first run this command:

$ ps fuxa | grep java

Then identify the PID of your Application Server:

someuser 11058  3.6 10.1 2607008 1660612 ?   Sl   May13 225:40  \_ java  -Dprogram.name=run-default.sh -server -Xms1536m -Xmx1536m -XX:MaxPermSize=192M -XX:PermSize=192M -XX:+UseParallelGC -Xloggc:gc.log -XX:+PrintGCTimeStamps  -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:-TraceClassUnloading -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/company/jboss-4.2.2.GA /lib/endorsed -classpath /opt/company/jboss-4.2.2.GA/bin/run.jar  org.jboss.Main -c default -b 0.0.0.0

Now, check the File Descriptors associated with this Process ID, by doing that we can identify which logs this process is writing to:

$ ls -la /proc/11058/fd | grep .log
l-wx------  1 someuser somegroup 64 May 17 14:44 1 ->  /opt/company/jboss-4.2.2.GA/bin/scripts/stdout_stderr_201105130805.log

Or you can just search for the text fragment you added to the log lines, assuming that you added a line that produces “###linkVar###” in the stdout, you can run:

$ find /opt/company -type f -print | tr '\n' '00' | xargs -0 grep -i ####linkVar####

And with that, we come to the end of another post.

I hope “zip -u” makes your life easier, if you have any cool tips for troubleshooting or productivity, please share with us in the comments section below.

Cheers!