What I do here is more module/component testing of threaded parts of an application. Testing a module without threads is way easier than one with threads. I use JUnit and standard concurrency mechanisms for testing. Logging (log4j) can be a big help during test development.

Pattern Alpha:

Basic asynchronous event checking using semaphores explicitly.
The general idea is the following: first add a Mocked Listener inline:

	final Semaphore s = new Semaphore(0);
fw.addListener(new IModificationListener() {
public void fileModified(File f, ModifyActions action) {
log.debug("got event: " + f.getAbsolutePath() + ":" + action);
Assert.assertEquals("just_edited", f.getName());
Assert.assertEquals(ModifyActions.MODIFIED, action);
s.release();
}
});

It is important to note that the AssertionFailedError from JUnit will not cause the test to fail, because it happens in a different thread! It is wise to do some logging.
After that:

	Assert.assertTrue("Real content change",
s.tryAcquire(interval * 3,TimeUnit.MILLISECONDS));

We test if we got the right call in time. You have to choose a useful timing. Note that this should still be fast, so my advice would be: Determine what timing is the lower limit that still works for you – choose 10x of that. Also document for other development what the expected run time is.

A similar example:

	@Test
public void testReceiveSend_XML() throws Exception {
final String testmsgcontent = "<msg>hello</msg>";
final Semaphore s = new Semaphore(0);
msgService.registerReceiveMessageListener(
new IMessageReceiveListener() {
@Override
public void receivedMessage(UserId from_userid,
String content) {
// do some more checks here
s.release();
}
});
msgService.sendMessage(testUser2, testmsgcontent);
Assert.assertTrue(s.tryAcquire(2, 5, TimeUnit.SECONDS));
}

Pattern Beta:

more advanced checking of one or more handlers with a state chain.

I use this Counter (java, 1 KB) class with the functions inc() and await(statenumber).

Consider the following example:
State 0. User 1 sends a message to User 2. The message is received by User 2 (State 1), User 2 replies. User1 replies again, the message is received by User 2 again (State 2).
The test constraint is that the test is successful iff State 2 was reached within a certain time.

	@Test
@Prerequisite(checker = TestEnvironment.class)
public void testReceiveSend_MultipleInteractions() throws Exception {
final String testmsgcontent1 = "Testmessagecontent1";
final String testmsgcontent2 = testmsgcontent1 + " >> really?";
final String testmsgcontent3 = testmsgcontent2 + " >> yeah, srsly!";
final Counter c = new Counter();
msgService2.registerReceiveMessageListener(
new IMessageReceiveListener() {
@Override
public void receivedMessage(UserId from_userid,
String content) {
c.inc(); // We reached our next control check
if (c.getValue() == 1) {
Assert.assertEquals(testUser1, from_userid);
Assert.assertEquals(testmsgcontent1, content);
try {
msgService2.sendMessage(from_userid,
content + " >> really?");
} catch (Exception e) {
log.error("", e);
Assert.fail("Unexpected exception");
}
} else if (c.getValue() == 2) {
Assert.assertEquals(testUser1, from_userid);
Assert.assertEquals(testmsgcontent3, content);
} else {
Assert.fail("Unexpected call");
}
}
});
msgService1.registerReceiveMessageListener(
new IMessageReceiveListener() {
// practically a echo service
@Override
public void receivedMessage(UserId from_userid,
String content) {
try {
msgService1.sendMessage(from_userid,
content + " >> yeah, srsly!");
Assert.assertEquals(testmsgcontent2, content);
} catch (Exception e) {
Assert.fail("Unexpected exception");
}
}
});
msgService1.sendMessage(testUser2, testmsgcontent1);
Assert.assertTrue(c.await(2, 5, TimeUnit.SECONDS));
}

Pattern Gamma

So this was very simple so far, but we most likely don’t know the order of events that might come in. Also, I’d like the events to have names rather than numbers.

So what I really want is

• to enter a event in the queue: step(“myevent”);
• to wait until a step is reached no matter if it is exactly the next: awaitStep(“something”); This removes the step from the list.
• to assure that was all that was received (the list of events is empty): isDone()

This Tracer.java (java, 3 KB) class helps me with that.
It also allows you to check only specific events and ignore others you enter (don’t use isDone).

An example:

	@Test
public void testSocketMethod_accepting() throws Exception {
// ...
tfm.request(this.outrequest, new INegotiationSuccessListener() {
@Override
public void failed(Throwable reason) {
t.step("clientside negotiation failed");
Assert.fail();
}
@Override
public void succeeded(IFileTransfer ft) {
t.step("clientside negotiation succeeded");
Assert.assertEquals(filename, ft.getFileName());
// make the variable available
TestSocketFileTransfer.this.fileTransfer = ft;
}
});
Assert.assertTrue(this.t.awaitStep("clientside negotiation succeeded", 1,
TimeUnit.SECONDS));
// it is assured that fileTransfer is available now
while (!this.fileTransfer.isDone()) { // This is a form of waiting in the main thread because this might take longer
try {
log.debug(this.t.toString());
Thread.sleep(40);
} catch (InterruptedException e) {
// best effort, doesn't matter
}
}
Assert.assertTrue(this.t.await("getting content from fs,started on server", 40,
TimeUnit.MILLISECONDS));
log.debug(this.t.toString()); // print what else is in the Tracer
Assert.assertTrue("timeout", this.t.isDone(40, TimeUnit.MILLISECONDS));
}

As far as I know, there are no simple libraries for handling threaded testing, I don’t know if that is possible in a very simple yet flexible way.
Threaded testing is quite verbose (as you see), but once you saw these constructs, it is easier to read.

I welcome your postings or links to other approaches in threaded testing.

PS: My experience in threaded testing comes from working at Porsche Informatik and two semesters of a Software Engineering project for the university that handles project folders (modifying listeners for deleting/changing files) and transferring content over the network (XMPP, sockets, thus message and other state notifications). If you are interested, the project, called Jake will be release within a month.