If you have some experience with coding Java, have a strong interest in science and the technologies we use, thrive in chaotic open environments, and would like to work in one of the world’s best cities to live, here are the instructions for applying.

You can meet me at BOSC and ISMB next month. If you have any questions, contact me!

The plug-in is installed by default in Protege, but needs to be enabled via Project > Configure > Tab Widgets > PromptTab.

Here’s a screenshot showing differences in the UniProt “core” ontology from release 10.5 to 11.0.

For the comparison, the taxonomy.rdf.gz file from UniProt release 10.0 was used. This file is about 127M large (uncompressed). The machine on which the comparison was run is a (slightly obsolete) Itanium machine, with plenty of RAM.

Raptor

The first tool I tried was Raptor (1.4.14). After the usual configure — make — make install the conversion can be run like so:

zcat taxonomy.rdf.gz | rapper -e -o ntriples - file://taxonomy.rdf.gz# > taxonomy.nt

This completed in 38.9, 38.8 and 38.9 seconds (in subsequent runs).

The -e flag turns off validation. This doesn’t seem to have a measurable impact on performance, but is necessary to avoid erroneous “Duplicated rdf:ID value” errors (at least in another, larger file).

The data is decompressed on the fly to save time (and disk space).

Jena

Next I tried Jena (2.5.2). After adding all the jars to the classpath, the conversion was run like so:

zcat taxonomy.rdf.gz | java jena.rdfparse -b file://taxonomy.rdf.gz# -x - > taxonomy.nt

This completed in 2:38, 2:40 and 2:38 min.

This is too slow if I wanted to convert the entire UniProt RDF data set within reasonable time, but at least I got a (correct) warning about a bad URI that I hadn’t been aware of…

The JVM is JRockit (5.0 R27.1) with default parameters (I tried some variations such as adding -Xgcprio:throughput, but didn’t see any significant change).

Rio

Last, I tried Rio (1.0.9), another Java parser. Rio doesn’t seem to include a command line tool for conversion, but it’s not a lot of code:

import java.io.IOException;
import org.openrdf.model.Resource;
import org.openrdf.model.URI;
import org.openrdf.model.Value;
import org.openrdf.rio.Parser;
import org.openrdf.rio.StatementHandler;
import org.openrdf.rio.StatementHandlerException;
import org.openrdf.rio.ntriples.NTriplesWriter;
import org.openrdf.rio.rdfxml.RdfXmlParser;

public class Converter
{
	public static void main(String[] args)
		throws Exception
	{
		Parser parser = new RdfXmlParser();
		final NTriplesWriter writer = new NTriplesWriter(System.out);
		writer.startDocument();
		parser.setStatementHandler(new StatementHandler()
		{
			public void handleStatement(Resource s, URI p, Value o)
				throws StatementHandlerException
			{
				try
				{
					writer.writeStatement(s, p, o);
				}

				catch (IOException e)
				{
					throw new StatementHandlerException(e);
				}
			}
		});
		parser.parse(System.in, args[0]);
		writer.endDocument();
	}

}

zcat taxonomy.rdf.gz | java Converter file://taxonomy.rdf.gz# > taxonomy.nt

This ran in 49.8, 49.5 and 50.0 seconds.

Using buffered readers or writers seemed to decrease performance slightly, so I assume the streams are already being buffered.

Conclusion

The conversion can be done fastest with Raptor. Rio is the best choice if you need to set up a platform-independent procedure (e.g. integrated into an Ant build). Jena is best if you also need to check the data :-)

Using aria2 — a command line download client that supports the Metalink standard — you can do:

aria2c ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/RELEASE.metalink

This will retrieve the data from all available mirrors in parallel, piece the files together, and verify the transferred data with checksums.

If you prefer graphical user interfaces, there is another tool called wxDownload Fast. Be sure to use the latest version (0.5.5) — previous versions wouldn’t do parallel downloads from some servers due to a small bug.

RDFRoom is a graphical RDF tool. It seems to work quite well (it’s written in Python) and performance is acceptable, at least for smaller graphs. Here’s a screenshot:

As you can see, I’m just deleting some outdated data…

After going through all the different terms that people use when talking about data (some of these terms have fallen out of fashion, others are still in use), Kent points out inconsistencies and limitations of the relational and hierarchical data models:

the data processing community has evolved a number of models in which to express descriptions of reality. These models are highly structured, rigid, and simplistic, being amenable to economic processing by computer. [...] Some members of that community have been so overwhelmed by the success of a certain technology for processing data that they have confused this technology with the natural semantics of information.

Of course Kent is full of understanding for those poor, misguided souls:

The builders and users of today’s commercial systems quite justifiably want to avoid cluttering their systems with anything that may impair efficiency and productivity. The argument that this new approach will make the overall management of data more productive in the long run has yet to be convincingly demonstrated to them.

He predicts that data integration will be the killer application for a more sophisticated data model:

The need for a more descriptive model will only gradually achieve general recognition. It will come from the headaches of trying to crunch together the diverse formats and data structures used by growing families of applications operating on the same integrated data base.

Kent then goes on to outline an ideal data model (graph-based). Nevertheless he manages to remain realistic:

Perhaps it is inevitable that tools and theories never quite match. There are some opposite qualities inherent in them. [...] Thus the truth of things may be this: useful things get done by tools which are an amalgam of fragments and theories. Those are the kinds of tools whose production and maintenance expense can be justified. Theories are helpful to gain understanding, which may lead to the better design of better tools.

[...] current ad hoc IR systems are not able to retrieve our example sentence when they are given the query ‘yeast cell cycle’. Instead, this could be achieved by realizing that ‘yeast’ is a synonym for S. cerevisiae, that ‘cell cycle’ is a Gene Ontology term, that the word ‘Cdc28′ refers to an S. cerevisiae protein and finally, by looking up the Gene Ontology terms that relate to Cdc28 to connect it to the yeast cell cycle. Although this will not be easy, we see this form of query expansion as the next logical step for ad hoc IR.

One of the interesting features of this search engine is that it attempts to cluster results by topic. This seems to work quite well for medical conditions (e.g. “stroke”), but is less suitable for distinguishing words used in different contexts (e.g. gene names from words). The ranking also needs some work, and a search for “uniprot” results in “Did you mean uniprost”…