On Static Code Analysis

Introduction

For a long time I thought that the static code analysis for a Java project was a waste of time. We know how to code correctly and nicely, we have the compiler, we have the unit tests, integration tests etc. why need to bother with something else?

When I started to get interested in this domain, many people in my team kept telling me about Sonar. I looked into it and I found it interesting how this tool manages to put together all sorts of aspects of the code quality. And then I dug deeper to see how the guys are actually using it. Because it’s cool to see the numbers and charts, but it’s even better if you act on them. And there I realized that the tool is somehow misused – people kept looking at those numbers, talked to their manager about the increasing “technical debt”, but still doing nothing, because the delivery of the project on time always took priority. And the debt kept increasing until people completely lost interest in the tool because it was too depressing.

My first experience with PMD, Findbugs and Checkstyle

I was amazed to see that finally there are mainly 3 tools that are used by everybody: Checkstyle, PMD and Findbugs. They are also ultimately used by Sonar as well.  Furthermore, the tools seem to overlap in different areas, so in the you have to use the three of them and make sure that the common areas are configured similarly.

Once I figured out what is the purpose of all three, I went back to my original problem – how to make use of them so that my colleagues would not get back to the same behavior: look at the number of detected problems -> see it’s too big -> do nothing.

The solution was to activate all the three plugins via Maven. The rules for all three were put in an external artifact, so people would add it in their pom file. Once activated for a project, the build is broken when an error is found. So this time people HAVE to fix the problem like any compilation problem, not to procrastinate it for a “stabilization phase”, (well it’s not 100% true, because you can still disable each of the tools for a given part of the code – maybe this should not be possible as well).

Everytime I suggested to the guys to enable it for some old project, people were afraid that it might take ages until the code gets compliant to all the rules. So to have a better idea of how long this endeavor may take, I did it for the open source project I work on – ST-JS.

  • Findbugs: 42 errors – 2h10 to fix
  • Checkstyle: 133 errors – 1h25 to fix
  • PMD: 311 errors – 11h15 to fix

So in total: 14h50 (almost two days)

Just to give you an idea of the code size of the project (SLOC using the linux command sloccount) : 7757

Even though the project has decent test coverage, the experience allowed me to find two corner cases that were potential bugs (using Findbugs) and some other dangerous situations.

Fixing the problems signaled by PMD  was the most painful of all, especially the part regarding the Cyclomatic Complexity. I confess we were having even 18 complexity for a given method (!).  Even though the parts of code seemed small and not very complex , several of these blocks together in the same method looked quite ugly. But that’s why I believe that you need something that would break your build, because we didn’t go to 18 in the first version, but incrementally by fixing one bug after another (or adding a new small feature).

When you fix these bugs, it’s important to have a good test coverage because when you start refactoring to lower the complexity, it’s incredibly easy to break the code! So be careful!

Now hoping that I convince you that it’s a good fight to take time to add static code analysis to your projects, let’s explore some thoughts on how we can be even better at it.

A word about ST-JS

We built ST-JS to add compile time error detection to JavaScript – so you write your code in Java that’s translated in JavaScript. But with static code analysis we can go even further, much further than where similar tools for JavaScript would take you.

Bug By Example

Findbugs and PMD are great to help you to asses the current status of a project and keep a certain level of confidence in your code’s quality. But in my opinion they both fail in making the life easy for the teams who’d like to continuously improve the quality of their code in a sustainable manner.

Creating new rules in Findbugs seems very difficult. In PMD it seems a bit easier using the XPATH framework, but it’s still quite hard and unnatural. Additionally even existing XPATH rules have problems because they match on class’ simple name instead of the full name. And generics are not well taken into account (try having for example a class called Statement or Thread in any package you want 🙂 )

While looking at the PMD documentation, you see that most of the rules come with examples. So I came up with the following idea: why not describing the rules by using examples? (bug by example).

The idea is to write compilable code snippets using a specific library to describe the bugs and then to detect those patterns in your code.

import static org.bugby.wildcard.Wildcards.someTypedValue;
import org.bugby.annotation.BadExample;

@BadExample
public class DontCallThreadRun {
 public void someCode() {
 someTypedValue(Thread.class).run();
 }
}

Making easy for the developers to create new rules is like creating unit tests for code yet to be written! Can you imagine delivering your libraries together with some rules that would statically check if it’s wrongly used!

Currently there are around 700-800 rules in Findbugs + PMD. But I believe that if teams and the community contribute to the rule database it shouldn’t be unusual to see enterprise projects with 5000-10000 rules enabled!

The Bugby project is still in its inception, proof of concept phase. Your help to move faster would be greatly appreciated!

Advertisements

ST-JS has a new website!

It’s been a while I haven’t updated this blog! Even though when I started it I was full of hope to be able to update it at least once a month, I was finally caught up with my work and with continually improving the project.  But ST-JS steadily increased its awareness and new users try it each and every month.

We’re preparing ourselves for the next big steps – to allow our users to contribute with bridges and libraries so that we can offer a much larger choice of bridged libraries to the users. Currently we’re only offering jQuery (even that bridge needs update!), but we had users using ST-JS in combination with Dojo, Prototype and AngularJS. We also need to follow the HTML5 API.

More we have the ambition to finish the very aggressive minifier that Nicolas started – it gets a bit easier when you have more information about the fields/methods visibilities and types, but still it’s not trivial.

One other interesting challenge we have is to write the reverse engineering application – the application will take existing JavaScript files and will try to guess the data types and write back the corresponding Java classes. The idea is to facilitate the adoptation of ST-JS and make the life easier for people that are looking for a solution like ST-JS. But for this … well I may need to take again a 3-weeks holiday in the south Spain – like it happened when well this started 🙂  Or attract other contributors motivated to help with the numerous challenges ST-JS proposes.

But the main reason I’m back updating the blog is to announce our new website created by Nicolas. Don’t hesitate to take a look  and let us know what you think!

STJS and jQuery

We were already using jQuery when we decided to build STJS. GWT handles the cross-browser issue by generating separate code for each target platform. We believe that it’s easier to leave this task to external libraries such as jQuery.

To be able to use jQuery in Java (with STJS) you need to have what we call a “bridge”. This is a Java code composed mainly of Java interfaces and empty static methods that mimic the Javascript functions. The static methods have no body or throw UnsupportedMethodException.

The jQuery core bridge

The bridge to the jQuery core functionality is found in the org.stjs.javascript.jquery.JQueryCore interface. Here is an excerpt from this interface.

@SyntheticType
public interface JQueryCore<FullJQuery extends JQueryCore<?>> {
 public FullJQuery addClass(String className);

public boolean hasClass(String className);

public FullJQuery removeClass();

...

The interface is annotated with @SyntethicType as this bridged type does not exist in the corresponding Javascript code (no JS function called JQueryCore).

One of the most handy features of jQuery is its ability to chain methods together, so it was important for us to represent this feature in java. So as you can see, the most of the methods return object of the type FullJQuery, which is a generic parameter that has to conform to the JQueryCore interface. This element is the key point that allows extending jQuery via plugins, as I explain later.

To make the usage of jQuery API more precised in Java, we tried to propose methods with the signature as strict as possible – so we tried to avoid Object or Object … parameters wherever possible. This was quite a challenging task  – and we didn’t get it quite right from the first try 🙂 as the library’s documentation is not always precise regarding the parameters’ types. Here is a short excerpt:

public Object css(String propertyName);
public FullJQuery css(String propertyName, Object value);
public FullJQuery css(Map<String, ? extends Object> propertyMap);

Even though when you write regular (non bridge) Java classes for STJS overloading is forbidden, for clarity purposes, when you write bridges this is allowed. As you can imagine, in our case all the three methods refer to the SAME Javascript function, but where the overloading is handled internally by the function.

function( elem, name, value ) {
 return value !== undefined ?
 jQuery.style( elem, name, value ) :
 jQuery.css( elem, name );

Plugins

The problem is that jQuery has a design that might seem unfamiliar to most of Java developers. I don’t know of a Java library where your library’s plugins add methods to your classes! Most of the time,you have an API and accompanying SPI, such as javax.crypto.Cipher and javax.crypto.CipherSpi or javax.imageio and javax.imageio.spi, and the plugins are added by implementing the SPI, whereas clients still use the API. In fact the Java language itself is not yet ready for this type of “extension”. The concept is called “Extension Methods” and the C# implementation is the closest to this concept. The Java implementation that should arrive in Java 8 is not quite helpful for this type of design.

One of the issues of Javascript in general and of jQuery in particular is the heavy overloading of the methods (especially for plugins). The same method can be used for activating the plugin, setting options or more unexpected – call methods and return values. Even though we tried to “strongly-type” the interface by providing several method signatures for the different plugins, there is still the problem of the return type (and additional parameters) that changes depending on the first parameter called “method name”. Here is an example:

@SyntheticType
public interface Draggable<FullJQuery extends JQueryCore<?>> {
 public FullJQuery draggable();
 public FullJQuery draggable(DraggableOptions<FullJQuery> options);
 public FullJQuery draggable(String methodName);
 public Object draggable(String option, String optionName);
 public FullJQuery draggable(String option, DraggableOptions<FullJQuery> options);
 public FullJQuery draggable(String option, String optionName, Object value);
}

One of the problems we had when we used jQuery prior to STJS was that we never knew by heart what options a plugin offers or what parameters you can access in the event handler. That’s why, instead of proposing as a parameter a simple Map for the options we went further and we strongly-typed that part too. So here how it looks (partially) the options parameter and the ui object for the Draggable plugin:

@SyntheticType
abstract public class DraggableUI<FullJQuery extends JQueryCore<?>> {
 public FullJQuery helper;
 public Position position;
 public Position offset;
}

@SyntheticType
public class DraggableOptions<FullJQuery extends JQueryCore<?>> {
 public boolean disabled = false;
 public boolean addClasses = true;
 public Object appendTo = "parent";
 public String axis = "false";
 public String cancel = ":input,option";
 public String connectToSortable = "false";
...

The jQuery core and the UI plugins are all brought together in the interface JQueryAndPlugins:

@SyntheticType
public interface JQueryAndPlugins<FullJQuery extends JQueryAndPlugins<?>> extends JQueryCore<FullJQuery>, //
 JQueryUI<FullJQuery>,//
 Accordion<FullJQuery>,//
 AutoComplete<FullJQuery>,//
 Button<FullJQuery>,//
 Datepicker<FullJQuery>,//
 Dialog<FullJQuery>,//
 Draggable<FullJQuery>,//
 Droppable<FullJQuery>,//
 Progressbar<FullJQuery>,//
 Resizable<FullJQuery>,//
 Selectable<FullJQuery>,//
 Slider<FullJQuery>,//
 Sortable<FullJQuery>,//
 Tabs<FullJQuery>//
{
}

This interface is intended to be further subclassed by the users who wish to add other jQuery plugins. If you plan to use only the plugins listed above you can use the JQuery interface that is a shorter name for the previous interface:

@SyntheticType
public interface JQuery extends JQueryAndPlugins<JQuery> {//
}

Writing your own plugin bridge

It may happen that you need to add a new jQuery plugin in your code (let’s call it MyPlugin). Following the same rules as for the UI plugins you’d write the interface representing your plugin:

@SyntheticType
public interface MyPlugin<FullJQuery extends JQueryCore<?>> {
 public FullJQuery myplugin();
 public FullJQuery myplugin(MyPluginOptions<FullJQuery> options);
 public FullJQuery myplugin(String methodName);
 public Object myplugin(String option, String optionName);
 public FullJQuery myplugin(String option, String optionName, Object value);
}

Then, wherever you previously used the JQuery interface you should use the MyJQueryLib interface defined as follows:

public interface MyJQueryLib<FullJQuery extends MyJQueryLib<?>>
 extends MyPlugin<FullJQuery>,
 JQueryAndPlugins<FullJQuery>{
}

To make it easier (no need for cast) you can re-write the global $ method to return your new JQuery interface as it’s described in the next paragraph.

The global methods and object

There are basically two ways to access the jQuery functionality from Javascript: via the $ global object or via the $ global method (I assume the $.noConflict is not used).
The global object gives you access to data and methods not necessarily linked to a DOM node, like $.browser or $.inArray().
The global method $()  uses a selector to build a list of DOM elements to which subsequent calls will be applied.

In STJS, the global  jQuery method and object are defined in the class: org.stjs.javascript.jquery.GlobalJQuery. They are defined as follows:

abstract public class GlobalJQuery {

public static GlobalJQuery $;

public static <FullJQuery extends JQueryAndPlugins<?>> FullJQuery $(String path) {
 throw new UnsupportedOperationException();
 }

...

}

The GlobalJQuery object contains all the methods and fields that can be accessed with $. like $.browser and $.inArray.
As you can see, the $ method returns a jQuery wrapper object that provides you all the methods of  jQuery core and UI plugins. If you decide to add new plugin and if you want to access to the newly added methods without casting, you may need to define your own GlobalJQuery inheriting from the STJS one:

@GlobalScope
abstract public class MyGlobalJQuery extends GlobalJQuery {
 public static MyGlobalJQuery $;

public static MyJQuery $(String path) {
throw new UnsupportedOperationException();
 }

..

}

Even if you don’t add methods to your MyGlobalJQuery instance you still have to re-define the $ global object in order to be able to statically import in your code the $ object and method

import static my.package.MyGlobalJQuery.$;

I hope this article gave you some insight on the challenges we had implementing this bridge. It may hopefully serve you as a starting point for implementing a bridge to other similar Javascript libraries.

Next article will describe how to implement a jQuery widget using STJS.

Strongly-Typed Javascript

I had the idea of building a Java to Javascript generator in the summer of 2011 when I saw the time we spent on writing, debugging and fixing Javascript code compared to the similar task in Java. Let alone the very poor support from our preferred IDE – Eclipse. We obviously had a look at GWT at the time, but it looked way too complicated for what we needed. I asked around the people I knew who had an experience with GWT and unfortunately they confirmed my feeling. So I spoke to my old colleague and friend Eyal about the idea, and to my great surprise not only he thought the idea is worth something, but he agreed to help me developing it. So here how Strongly-Typed Javascript was born! The name itself was supposed to be temporary – let’s develop it and we’ll find a name later 🙂 But the acronym finally stuck, STJS being quite easy to remember.

Since we launched STJS, Google launched the Dart language. But we still believe that it’s not exactly the same direction. In fact Google themselves are not quite sure of how these two technologies fit together. And after I read the article http://www.javacodegeeks.com/2012/01/gwt-pros-and-cons.html I believe that other developers share our feeling that GWT maybe not be the final answer to web development for Java developers.

Three months after the first public release, we received quite encouraging messages from the developers who “dared” to use it. After the first article on TheServerSide, I believe more in-depth presentations of the technology may be of interest and a blog can be a good starting point.

%d bloggers like this: