I uploaded a jar file containing a simple test applet to Google Docs and referenced it like so:

<applet
  width="440" height="60"
  code="JavaVersionDisplayApplet.class"
  codebase="http://docs.google.com/"
  archive="uc?id=0B9ANnqvnRtroOTM5YWE3YmMtMGZkMy00ZGE5LWE2NWUtOGZhNzk5MzA2NzBk">
</applet>

The result:

James A. Whitaker (Microsoft) on The Future of Testing: Crowdsourcing testing (see e.g. uTest) is the simplest way to get an application tested on a large number of different platforms. Such services should also allow testers to focus more on designing rather than executing (manual) tests. Virtualisation could address the issue of hard to reproduce bugs by capturing the state of the machine as part of bug reports. Game testers use maps to highlight areas in game worlds that have been thoroughly tested. Similar visualizations could help people testing normal software what areas of the application need more testing.

Elfriede Dustin and Marcus Borch (IDT) on Advances in Automated Software Testing Technologies: There are tools that can help generate random test data. The speakers spoke in great length about generating test code from customer supplied spreadsheets. Not sure if anyone in the audience understood what the advantage of this approach is — I didn’t.

Andres Almiray (Oracle) on Boosting Your Testing Productivity with Groovy: This was more about the advantages of Groovy in general than testing, though towards the end of the talk the speaker did show how to create mock objects (using MockFor/StubFor) and how to use DbUnit in Groovy (more or less the same way you’d use it in Java). Oh boy this guy really did drink the Kool-Aid…

Markus Clermont and John Thomas on Taming the Beast: How to Test an AJAX Application: The speakers did their best to keep he audience awake in this after-lunch talk. This got a bit tiring after the tenth “raise your hand if…”, but they had an excellent point: The amount of large, end-to-end tests (written e.g. with Selenium) should be minimized in favor of “medium”-sized tests that test interactions between specific components (one level deep).

Matt Wood (Sanger Institute) on The New Genomics: Software Development at Petabyte Scale: This was perhaps the most interesting talk, though it didn’t really go into testing at all!

Marc-Elian Bégin (Six2) on Using Cloud Computing to Automate Full-Scale System Tests: The case was made that Amazon EC2 is of great use for (system-level) testing. The speaker predicts that paravirtualization (e.g. with Xen) will increasingly be replaced with full virtualization (e.g. KVM), even though latter requires special hardware support (built into most newer processors?).

Vishal Chowdhary (Microsoft) on Practicing Testablity in the Real World: Good explanation of the basics of writing testable code, though perhaps a bit too basic for the audience that was present.

So far there are 11 books, from Jess in Action (published July 2003) to jQuery in Action (published February 2008). I’m somewhat surprised that there was no announcement (so far) on either company’s web sites… In any case I’m hoping that there is more to come!

Sessions

These are the sessions I attended:

• TestNG with Code Coverage (Howard Lewis Ship)
• Groovy: Metaprogramming (Scott Davis)
• Professional Java UI Development with Eclipse RCP (Brian Sam-Bodden)
• The Busy Java Developers Guide to Concurrency (Ted Neward)
• Code Metrics & Analysis for Agile Projects (Neal Ford)
• Building Web Apps with Spring OSGi (Craig Walls)
• The Busy Developer’s Guide to Debugging (Ted Neward)
• The Busy Developer’s Guide to Monitoring (Ted Neward)
• The Busy Developer’s Guide to Scala (Ted Neward)
• The Busy Developer’s Guide to Hacking on the JVM (Ted Neward)

Notes

Test coverage should at least be 85% as there is often a significant number of bugs that are flushed out when you cross that threshold. But if you don’t aim for 100% coverage, there is the risk that you start copping out on testing the most difficult code!

When using the EclEmma Eclipse plug-in for gathering code coverage statistics, set up separate build output folders — otherwise unit tests are included in the statistics (even if the unit test source folder is not selected).

Cobertura generates somewhat better code coverage statistics than EMMA, but lacks an IDE plug-in.

Maven is a great idea, but using it for your project can be “like living with a dangerous psychopath” — it behaves when showing it to other people, but you never know when it will turn on you….

Ant has a presetdef task that allows you to create an alias for a task, and set default attribute values.

TestNG can be used both for unit and for functional tests. Test “groups” can distinguish the two (quirk: can’t directly select all tests that are not in a group). TestNG creates only one instance of each test class. TestNG can run tests in parallel (either at the class or at the method level). There is an @DataProvider mechanism that can return either an array of test data or an iterator. TestNG can run JUnit tests! Can also run Selenium RC from TestNG — check the Tapestry test code.

New Groovy language features don’t require new Java versions. This may be attractive for people who are stuck with old versions of Java. Groovy even allows addition and modification of all methods at runtime via its ExpandoMetaClass, e.g. String.metaClass.foo = { -> delegate.toUppercase() }

A JVM may map threads n:m at the OS level.

Schedulers may do “priority inversion” when a low priority thread holds a lock required by a high priority thread.

Build-specific deltas are more interesting than absolute numbers when looking at code metrics.

The “teddy bear” debugging technique: Explain the problem to a teddy bear! Often you’ll realize what the problem is before you need to go bother a colleague…

Should declare unchecked exceptions.

-XX:OnOutOfMemoryError=cmd sets a command that is run when we run out of memory.

-XX:TraceClassLoading shows where classes are loaded from.

The extconflict tool (shipped with Sun’s Java) detects duplicate classes in lib/ext.

Heisenberg principal: Monitoring a system always influences it (e.g. garbage collection activity due to jconsole connection).

jconsole has a scripting plugin. Also consider using Groovy for scripting tasks.

Scala is an (impure) functional language. Functional languages are great for concurrent programming (immutable objects, use of “actors” etc). But Scala won’t be ready for production use this year (or even next year). Get the book!

Reflection can be used to read (and set) private fields, even in core classes such as java.lang.String! Going one step further: We can provide our own String implementation (use -Xbootclasspath=...).

Comments

Most of the talks were at a level of detail that if you happened to have read a book on the topic you weren’t likely to learn a lot. Nevertheless you were bound to pick up some useful stuff as (most of) the speakers were clearly not only quite experienced but also great speakers. Ideally you’d have the number one expert on each topic give the talk, but I realize that isn’t practical… Despite the “No Fluff” branding there was a lot of talk about how we all need to be “agile” etc. Nevertheless I think everyone managed to keep their feet on the ground, and in any case their were sufficient technical talks. Not to mention the food and free stuff :-)

First I tried using POI-HSSF (a well-known Apache project). The code is reasonably straightforward — though I had to get it from tutorials on the Web as I couldn’t find much in terms of documentation on their site, and the API isn’t exactly self-evident:

POIFSFileSystem fs = new POIFSFileSystem(in);
HSSFWorkbook wb = new HSSFWorkbook(fs);
HSSFSheet sheet = wb.getSheetAt(0);
Iterator rows = sheet.rowIterator();
while (rows.hasNext()) {
    HSSFRow row = (HSSFRow) rows.next();
    Iterator cells = row.cellIterator();
    List<string> fields = new ArrayList<string>();
    while (cells.hasNext()) {
        HSSFCell cell = (HSSFCell) cells.next();
        switch (cell.getCellType()) {
            case HSSFCell.CELL_TYPE_STRING:
                fields.add(cell.getStringCellValue());
                break;
            case HSSFCell.CELL_TYPE_NUMERIC:
                fields.add(Double.toString(cell.getNumericCellValue()));
                break;
            default:
                log.warn("Unsupported cell type: " + cell.getCellType());
                break;
        }
    }
    process(fields);
}

This works fine — except that there doesn’t seem to be a way to keep the precision of numbers, i.e. if you don’t want to convert “4″ to “4.0″! Fortunately POI isn’t the only game in town. I ended up using JExcel API which has a much simpler API (and behaves the way I want it to):

Workbook workbook = Workbook.getWorkbook(in);
Sheet sheet = workbook.getSheet(0);
for (int i = 0; i < sheet.getRows(); ++i) {
    List<string> fields = new ArrayList<string>();
    for (int j = 0; j < sheet.getColumns(); ++j) {
        fields.add(sheet.getCell(j, i).getContents());
    }
    process(fields);
}

1. Should I disable hyper-threading (ht)?
2. How much does doubling the available memory improve performance?
3. Mirror, mirror on the wall, who has the fastest JVM of them all?

Method

The machine in question has a single 2.8GHz Intel Xeon processor with 2GB RAM and is running Linux (2.6.9-42.0.3.ELsmp).

The application is a web application with a large (several gigabytes) Lucene index and Berkeley DB for storing data. The main interest was the performance of the data import procedure, though I also had a brief look at the search and retrieval performance.

The following (Java 6) VMs were tested:

• BEA JRockit
  build R27.3.1-1-85830-1.6.0_01-20070716-1248-linux-ia32
• Sun Java HotSpot Server VM
  build 1.6.0_02-b05
• IBM J9 VM
  build 2.4, J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20070817_13537

Note that even though the machine has a 64-bit processor, I used the 32-bit versions of the JVMs as these are faster than (or the same speed as) the corresponding 64-bit versions (depending on whether the JVM is smart enough to always use 32-bit pointers whenever the heap size is small enough).

Some of the tests were repeated with some of the JVM parameters described by Henrik Stahl. But I never managed to beat the default configuration! So I ended up using only -Xms and -Xmx for all tests.

Results

Median response time (s) for running 1′000 queries sequentially:

Median response time (s) for running 1′000 queries from 10 parallel clients each:

Total time (h) for loading and indexing (in a single thread) all data (UniProt release 12.2):

Discussion

Hyper-threading can slow down single-threaded applications a bit (perhaps depending on whether the bottleneck is garbage collection, which can be done in a separate thread, or IO). But as soon as there are lots of threads there appears to be a clear benefit in keeping hyper-threading enabled.

More memory is better (surprise)… Doubling the memory in the system from 2 to 4gb (and increasing the heap size from 1 to 2gb) improved performance by about a third for some configurations. Doubling again however may not be worth the expense (diminishing returns).

The latest version of Sun’s JVM appears to be a bit ahead of the others… for now!

Nevertheless I was a bit taken aback when I discovered this sign at the entrance to our garage:

Moreover, the prohibition sign to the right contains a figure that resembles a hacker wearing a hat!

Microsoft-sponsored conspiracy, or misunderstanding?

Hint: I’m currently working in a French-speaking region.

If you have some experience with coding Java, have a strong interest in science and the technologies we use, thrive in chaotic open environments, and would like to work in one of the world’s best cities to live, here are the instructions for applying.

You can meet me at BOSC and ISMB next month. If you have any questions, contact me!

Here’s a 3D view of the source code for beta.uniprot.org:

I do wonder what all those lines and boxes mean, but it certainly is pretty!

And here we zoom in on one frequently used class:

Any questions? OK, maybe I’ll have to keep looking for the perfect visualization tool :-)

Here are the slides that I showed along with the demo I gave:

The workshop (which was really more of a series of normal talks than a real workshop) ended with a SOAP vs REST debate. The SOAP camp felt that their technology would in some unspecified way address interoperability issues. The REST camp on the other hand had plenty of war stories that showed that SOAP helped little with that but introduced a lot of problems. Of course no consensus was reached, but some of the participants volunteered to write a review paper that explains the different positions… Let’s see what comes out of that.