Comparison of Java ME unit testing frameworks
With the recent ME Framework open sourcing, and with all those discussions on what ME Framework can do for application developers (as opposed to TCK writers), I took a look at the APIs for popular JUnit-like unit testing frameworks for Java ME, to see what we can do better in our ME Framework.
Please note that this comparison is focused mostly on the API.
Popular Unit testing framework for Java ME:
All three frameworks are close to the original JUnit, with adjustments needed to support the Java ME Platform. In all 3 cases, a test class should extend a base class from the appropriate framework. And all typical JUnit helper methods are provided to the subclasses, e.g.: assertTrue, assertFalse, assertEquals, etc. (J2MEUnit provides the bare minimum of such methods, leaving quite some of them out).
JUnit standard hook methods are also present: setUp() and tearDown(), to setup test environment and to tear it down after the test is done.
So, in all 3 cases, developers who are familiar with JUnit, can start writing new unit tests for Java ME with fewer problems, since they basically know the core API.
The differences can be seen in areas where support for Java ME was added (one of the major problems is the lack of reflection in Java ME, while the original JUnit relies heavily on it). While the tests themselves look pretty similar for all three frameworks, the way the test suites are constructed differs.
1. J2MEUnit
With absence of reflection, J2MEUnit authors decided that test suites should be created manually, explicitly adding new tests to the test suite, like this:
aSuite.addTest(
new TestOne("testOne", new TestMethod() {
public void run(TestCase tc) {
((TestOne) tc).testOne();
}
})
);
The name of the test provided in the constructor, and the functionality of the test is enclosed in the anonymous inner class that implements the TestMethod.
Pretty heavyweight and not that simple, if you ask me.
2. JMUnit
JMUnit test class must call a two-parameters constructor of the base class, specifying the number of the “sub”-tests, and the name of the test.
public TemperatureConversionTest() {
super(4,"TemperatureConversionTest");
}
The test selection and execution is done via method test(int), that should be implemented by every test:
public void test(int testNumber) throws Throwable {
switch(testNumber) {
case 0:testfahrenheitToCelsius();break;
case 1:testcelsiusToFahrenheit();break;
case 2:testisHotter();break;
case 3:testisCooler();break;
default: break;
}
}
The base class will call this method with all int values, from 0 to number-of-tests.
In my opinion, this approach is less convoluted and more straightforward then J2MEUnit. JMUnit also provides better GUI on a device that shows the results.
JMUnit provides two versions, one is suitable for CLDC 1.0, and the another one is for CLDC 1.1.
Overall, JMUnit provides more extensive set of helper methods (asserts), with better GUI, and with more straightforward configuration.
Netbeans’ Mobility Pack has switched from J2MEUnit to JMUnit recently.
3. Sony Ericsson Mobile JUnit 1.0
The folks from Sony Ericsson did an excellent job on this, providing good extension to JUnit for Java ME, and with great documentation 
Sony Ericsson approach, it seems, was to provide experience as close to original JUnit as possible. In most cases, to port JUnit tests to Sony Ericsson unit tests is just a matter of replacing the base test class all the tests extend (from JUnit one to Sony Ericsson one.)
Couple of innovative things were done in this library. The absence of reflection is handled on Java SE side, where the tests are being analyzed and proper extra classes are created, so that developers are freed from manual work of specifying all their test names. Sony Ericsson’s version is clearly better here then the two alternatives (J2MEUnit and JMUnit).
The library also provides nice additions on top of typical JUnit API. For example, it not only provides setUp() method, but also setUp(MIDlet midlet), for those tests that would like to access the midlet that runs the tests.
The GUI on the device looks nice too, with tabbed interface for passed/failed tests. Sample project for WTK is also provided. And the basic code coverage can be calculated, which is an unexpected plus.
Integration with Ant, NetBeans and Eclipse is provided.
Overall, Sony Ericsson Mobile JUnit 1.0 is very well done and easily beats other alternatives, discussed above. They provide the best JUnit “emulation”, and resolve absence of reflection in innovative way simplifying developers life.
For additional information, take a look at the following article: “JUnit Testing Using Java ME JUnit Frameworks“.
April 5th, 2007 at 8:25 am
How about cldcunit?
http://pyx4me.com/snapshot/pyx4me/pyx4me-cldcunit/index.html
The only drawback of SE Mobile JUnit is that it is not open sourced so you can not customize or improve it for your own purpose.
April 5th, 2007 at 8:48 am
Good post Vladimir!
One thing I find hard to accept in JMUnit APIs is that the test class hierarchy uses MIDlet as its base class (Assertion extends MIDlet).
First, it ties all tests to the MIDP platform and they are not reusable elsewhere even if the code under test is not MIDP-specific.
Second, it limits test automation possibilities. MIDP spec restricts MIDlet creation: only AMS is allowed to call “new MIDlet()”. This means that the only way to run JMUnit test is to use AMS to manually start the test; one cannot write a custom launcher without changing JMUnit APIs.
The other annoyance with the MIDlet-based hierarchy is that one cannot build JMUnit tests into the existing MIDP applications which are MIDlets themselves.
Hopefully, JMUnit 2.0 (the next version, which is supposed to be a merger of J2MEUnit and JMUnit) will fix the hierarchy and make it independent of the application model.
April 6th, 2007 at 9:47 am
[...] Comparison of Java ME unit testing frameworks- Testing code on mobile devices? This is your page, man. [...]
April 26th, 2007 at 6:54 pm
arcoxia…
This is a great article. I am new to your blog and i like what I see. I look forward to your future work.thanks for the tips. arcoxia if i made this at the beginning all the things were much easier….
August 14th, 2007 at 4:59 am
What about MoMEUnit?
Its like J2MEUnit but with a better GUI
SE Mobile Unit is worst solution because you only can run test by DebugOnDevice and not directly on any device
I am waiting to J2MEUnit 2.0, really it sound pretty good
FFFidalgo
September 17th, 2007 at 8:22 am
Is anyone doing Test-Driven Development for MIDP? I’m interested in seeing these tools compared for how well they support the quick test-code-test-code rhythm you want when doing TDD. Having to manually add each test to the suite is painful. So is having to fire up the AMS (in the emulator or a device) to test a bit of application logic that has nothing to do with MIDP specifics. We’ve got some people coming onto our team that are new to TDD and I’m anxious to make the experience as painless as possible or we’ll have trouble getting them to buy in to it.
February 29th, 2008 at 3:46 am
The company that I’m at uses TDD for MIDP using JMUnit and a MIDP mock object framework that we wrote. I don’t come from a TDD background but my experience has been
1. I don’t find adding the tests to the JMUnit switch statement a pain but I’ve got into a rhythm that I like: (1) write the test, (2) add the test to the switch statement, (3) run the emulator, (4) check that the test fails, (5) do the implementation, run the test in the emulator again. Perhaps because I don’t come from another TDD background I don’t find steps (2) and (3) particularly onerous because I don’t have anything to compare against.
2. I find that I spend more time writing tests than application code but I’m happier with the application code that I produce; I spend less time coding and more time thinking (since I have to think about how to write testable code).
3. The MIDP apps that we write seem to have a lot of UI components and the application logic associated with actions for Forms, Lists, Canvases and even non-UI components may be common to several. When I find that I’m having to write a test that is similar to one that I’ve written for some other component I tend to extract a superclass, and then write tests for the super class so that I know that I don’t have to write boilerplate test code for the other classes that will extend the super class. I.e. I refactor more than I used. Naturally the unit tests give one a warm glow when they still pass after the refactoring.
4. When bugs are found outside of unit testing, I have been surprised at how easy it is to write short unit tests to replicate the bugs. TDD would appear to result in code whose testability goes beyond the tests written for the TDD.
I’m sure that none of the above is surprising to experienced TDD developers but as a Java ME developer who doesn’t come from a TDD background I found the test, code, test idiom great.
Finally, my feeling is that TDD in Java ME results in high quality code. In particular on my current project I have written 225 unit tests of which 221 were written as part of the TDD process. Outside of unit testing I found 3 bugs myself and 3 of the unit tests reflect those 3 bugs. A fourth bug was found during QA and the 225th unit test was written to replicate the bug.
Now for some caveats:
1. I’m not an authority on TDD.
2. Other people feel that writing testable Java ME code bloats it which may be serious if you’re building for old MIDP 1.0 devices. My feeling is that TDD allows for easier refactorings and that the refactorings can lead to significant code saving optimisations. But I have no metrics for either point of view.
July 8th, 2008 at 11:15 am
[...] So sánh và đánh giá các Unit test cho J2ME [...]
July 22nd, 2008 at 10:05 am
[...] a unit testing framework for Java ME it seems not much has happened. There seems to be basically three options and none of them seems very actively developed. I wonder [...]