Coding

xUnit and Fitnesse

2007-12-26T13:59:00.000Z

Fitnesse and NUnit.

One of the current arenas of competition in the testing arena is the comparison with xUnit style testing and Fitnesse style testing.

Fitnesse is good at allowing users to enter test cases so long as they conform to a programmed template. It also is good at presenting the results and what was tested in a presentable way, typically with a web page. The Fitnesse style frameworks I have come across are not good at summarising the results in a one page summary.
xUnit style testing aren’t good at allowing the users to enter test cases. However with frameworks like mbUnit, they are now quite good at allowing lots of test cases to share the same code. They are good at summarising the results, but bad a presenting what has been testing.
In my experience, it’s quite rare that end users actually enter the data in a Fitnesse test case. It’s the programmers who use a Fitnesse style interface to present their test cases to end users. My conclusion is that it’s the lack of a presentation layer that is the problem with xUnit style testing.
It’s certainly good practice to use command-query separation when designing code in order to make it easier to test.
I propose that there is an ‘Assert. Command method added to the Framework. It takes a string as an argument and it acts as comment for the command sections. There are already asserts in place.

Code


namespace bank
{
using NUnit.Framework;

[TestFixture]
public class AccountTest
{
  [Test]
  public void TransferFunds()
  {
    Assert.Command (“Create source account”)
    Account source = new Account();
    Assert.AreEqual(0.00F, source.Balance);

    Assert. Command (“Deposit 200.00F in source account”)
    source.Deposit(200.00F);
    Assert.AreEqual(200.00F, source.Balance);

    Assert. Command (“Create destination account”)
    Account destination = new Account();
    Assert.AreEqual(0.00F, destination.Balance);

    Assert. Command (“Deposit 150.00F in destination account”)
    destination.Deposit(150.00F);
    Assert.AreEqual(150.00F, destination.Balance);

    Assert. Command (“Deposit Transfer 100.00 from source to destination”)
    source.TransferFunds(destination, 100.00F);
    Assert.AreEqual(250.00F, destination.Balance);
    Assert.AreEqual(100.00F, source.Balance);
 
  }
}
}

Output


TestFixture AccountTest                                         Status Count
Test         TransferFunds                                         Pass 1
Command         Create source account                                 Pass 1
Assert         AreEqual(0.00F,source.Balance)                         Pass 1
Command         Deposit 200.00F in source account                 Pass 1
Assert         AreEqual(200.00F,source.Balance)                 Pass 1
Command         Create destination account                         Pass 1
Assert         AreEqual(0.00F, destination.Balance)                 Pass 1
Command         Deposit 150.00F in destination account                 Pass 1
Assert         AreEqual(150.00F, destination.Balance)                 Pass 1
Command         Deposit Transfer 100.00 from source to destination Pass 1
Assert         AreEqual(250.00F, destination.Balance)                 Pass 1
Assert         AreEqual(100.00F, sort.Balance)                         Pass 1

So what does this presentation do? It presents a human readable output that can be presented to end users as to the details of what is going on in a test fixture. It’s the narrative of the test.

The Assert.Command passes when there are no failures between it and the end of the test or the next Assert.Command.

The count column lists the number of times the test has passed. Think about what happens when you have a loop. Typically here you would put a Assert.Command at the start of the loop. For example :-

Assert.Command (“Test all values in list”)

should probably the sort of comment that precedes each loop.
What then happens is each TestFixture, Test, Command and Assert can output their own little bit of XML that is then put together into a file at the end. You now have a narrative presentation that can be given to users, and they can read it, and make comments. They don’t need to understand the code underneath. In fact, the presentation is programming language neutral.

I’m not particularly happy with the presentation of the assertion details. Perhaps an expected value, against an actual value would be better. There is also the question of the message string in the assert.

Given the right presentation, where you can hide or expose the details, which isn’t difficult in an XML to HTML presentation, you end up with means of testing that does 99% of what Fitnesse does. It is just the question of allowing users to create their own test data.

Test Driven Development

2007-11-30T09:43:00.000Z

A blog article on testing versus correctness

What I find odd about TDD is this. Most tests are just the same as exercising the contract that you would write if your language supported DBC (design by contract).

i.e. They consist of setting up data conforming to the preconditions, passing the data to the routine, then asserting the post condition as the success or failure of the test.

TDD then just tests a very small subset of the possible combinations.

DBC on the other hand is the equivalent of TDD full time (until disabled). All the time you are running this style of testing on your code, for all the scenarios that you exercise.

There are three approachs then to automating the testing for this style.

For Eiffel there's a research project Eiffel Test Studio that generates test data automatically that tries to generate all valid combinations of data that meet the pre-conditions, to see if you break the post-conditions. It should also be possible to see which post-conditions are used, and maybe even of there are possible missing post conditions.

There's a microsoft reasearch project Pex doing a similar thing. Exercise the function with lots of data automatically, but watch the program as it executes. Try and set the data up so you exercise each path through the code. ie. Explore all the cyclometric complexity of the code.

Lastly, there is the old fashioned code analysis route, much helped by DBC.

ie. Most TDD should be automated. Programmers need to move beyond just writting the tests.

We need to move to the point where we are writting the difficult tests, not the easy tests.

Its also clear that programmers are starting to use programs to automate their development, just as the program they write are designed to automate or aid other people's work.

Dynamic Code Generation vs Reflection

2006-07-31T10:15:00.000+01:00

Extract from an interesting code project article.

Dynamic Code Generation vs Reflection

Introduction
This is an example of how to use the new DynamicMethod class in .net 2.0 as an alternative to Reflection. Specifically, this example shows how to instantiate an object and get/set properties and fields on an object. As an added bonus, this code sample also allows getting and setting private fields and properties as well as instantiating objects with private constructors.

Background
At my current company we have built our own Object-Relational-Mapper (ORM), similar to nHibernate. In order to do this we were using reflection to set the properties/fields of an object with the values we retrieved from the database using a DataReader. Also, one of the features that was important to us was the ability to have “Read Only” properties on our objects. This involved setting the private field with our data mapper, instead of the public property. Unfortunately, reflection is a bit slow, especially when using it to set private values.

Solution
Fortunately, along came .Net 2.0 and the DynamicMethod class. This is a new class that allows one to create and compile code at run time. This approach is much faster then using reflection (see the times on the screen shot above). The down side to this approach is that you need to write your dynamic code using IL (as opposed to c#). But, with only a few lines of code we were able to create the desired effect.

Unit Testing in Production

2006-06-06T12:01:00.000+01:00

There are some unit tests that should be run in a production environment. Sounds heretical, but wait a moment.

fsck is the Unix file system checker. It checks the file system and reports on errors. At heart it is just a unit test that looks for faults in the file system.

Quite a few unit tests can be written to do the same. For example, read a database and check that it is consistent.

Take a collection of bonds, price them and check the prices against an external source.

All of these tests work on large data sets and should be available in a system. They are particularly useful after an upgrade has been installed.

I have even written tests to check that a system has been configured in the correct way.

More on Fitnesse

2006-06-05T13:31:00.000+01:00

I've posted before on Fitnesse. Thinking more about it I think it does have one advantage over NUnit and unit testing.

When you are writing unit tests there are two ways that end up being used.

Development phase - test first, then code
Code coverage shows wholes in test strategy - code first, then write test
Bugs - code first. Then when detected write the test to show its existance and a test to show it has been fixed.

In almost all these cases the test gets written with inside knowledge of how the code works. In the development phase there is a lot of iteration between test and code. Similarly with the test second approach.

Fitnesse is different. The tests get written without any assumptions as to how the code is going to work. That is a significant difference.

Library Annoyances - Dot Net

2006-06-01T12:38:00.000+01:00

I'm getting really annoyed with some of the design of libraries, and in particular Dot-Net.

If we take the simple task of converting a string into an integer. Clearly we can expected strings to be passed in that cannot be converted into an integer. Dot-Net takes the approach of generating an exception. OK, acceptable practice so far. However, there isn't a method anywhere that enables testing of strings to see if they can be converted to an integer.

Public Function IsInteger(text as String) as Boolean

You have to write this function yourself.


Public Function IsInteger(text as String) as Boolean
    Dim i As Integer
    Try
       i = CDbl(text) ' or use the convert Class
       Return True
    Catch
       Return False
    End Try
End Function

To have to write this every time such that you can write code in this way


    If IsInteger(text) then
        ...
    Else
        ... ' Handle error
    End If

Is really really annoying.

Having an IsInteger function also means that writing unit tests becomes a lot easier.

It also means that you can attempt to put in place preconditions and post conditions in routines.

What is happening is that programmers haven't be brought up with Design By Contract where this sort of function is normal, because it will be used as a pre or post condition. This is one reason why the function should also be side effect free. If you know Eiffel then you will program this way automatically.

Similarly, they haven't thought about the need for functions in unit tests. Here you don't want to be messing around with exceptions unless you are testing an exception framework. You want clean code.

There is the implied connection between unit testing and DBC. Both are needed. Unit testing to exercise the code. DBC to test the code, when ever it is run, not just in a unit test.

NCoverExplorer

2006-06-01T10:29:00.000+01:00

I've been using NCoverExplorer as part of Testdriven.Net for some agile development. However, last night I found a bug. I sent the stack trace off to the support address. Grant writes back quickly. I send him some information about how to replicate the problem. New version is released within 2 hours of the initial bug report.

Thanks Grant. Appreciated.

A simple object DB

2006-05-31T17:23:00.000+01:00

Create a database with a schema like this relationship diagram.

Now we can easily create the following application.

When a trade message arrives, we store it in the XML table. At the same time, we run an XSLT style sheet on the message. This generates some of the fields in the Message table. In particular it generates the type of message, the time stamp, and the key. We then update the message table with a new row. This update method has to be a little clever in that it needs to work out if there is a row already in place with the same key. If so, it sets the version number to the next in sequence If not, it sets the message number to 1.

That in itself is sufficient to act as a message store of free form XML. Notice there isn't any normalisation of the data. The XML can change, and the application carries on working. At the same time, you don't have to analyse the data and normalise it in order to get it into the database. For something like a swap transaction that would be a lot of work.

So storage is easy. What about retrieval? Well, here we again can have an easy strategy. We create a set of keys to the data and store them in the key table. To generate the keys, we resort to the same method used to generate the key on the Message table. We take the XML, pass it through an XSLT style sheet, and have the
style sheet output just the keys. A small amount of programming and we have the keys generated.

Now we can retrieve by keys.

One likely scenario is the need to add extra keys. Again it isn't a problem. We have the original message. We need to iterate over the Message table. We clean out the old keys. We take the XML message and pass it through the new XSLT style sheet, and generate the new keys and store them.

As a pattern this is a very cute idea.

1. Nothing is specific to one particular application.
2. XSLT is the right approach for manipulating XML
3. The schema copes with changes in requirements without the need to be modified.
4. The pattern removes the need for lots of analysis to get the data into a persistent store.

The split between Message and XML tables is to speed up access by removing the CLOB data to the XML table. Quite often you will not need to query the XML table.

An extension is indicated by the archived flag. You may want to age data and even move it off into an archive table. Alternatively set up the indexes with the archived flag as part of the key to speed up access.

One use is as a message store for systems that don't support idempotent operations

Another is as a message archive to keep records of messages over the long term. Just attach an adapter onto a message queue and store the messages.

Use of the version number also means you can have a full record of the messages.

Version could be made a key, but the problem here is that you often just want the latest version. It is easier and quicker having this as part of the message table.

Fitnesse versus Unit Testing

2006-05-30T19:31:00.000+01:00

My feelings on Fitnesse are as follows. I don't think users want to use html to design tests. Users really want to see the results of unit tests presented in a way they can understand. The easiest mechanism for them to use is to see the results as html on a web page, and such that they can see or trust that the tests have been run.

Unit tests are not very good in this respect. There is no standard way for test inputs, test outputs, and the result of the test to be presented back to the user.

Given this was in place, I doubt that Fitnesse would be an attractive option.

So as a guess, it would work something like this in NUnit


OutputTestName     (name)
OutputTestParameter(name,value)
...
OutputTestResult   (name,value)
OutputTestAnswer   (boolean)
...

Output should be in the form of XML, with a style sheet and optional XSLT sheet to transform to XHTML.

I have seen people try and use Fitnesse as a precondition tester for the system as a whole. For example, has all the requisite static data been set up. Fitnesse can test to see if known static data has been set up. However, if you provide a report in the system for displaying the static data, you have a much more useful result.

The real test is a reconciliation test. Does the data in a particular system reconcile against a gold standard of static data.

ie. So far in my limited experience of Fitnesse, all the testing problems are better solved using standard regression testing techniques, particularly if there are some useful extensions provided do the testing frameworks.

NUnit - A deficiency

2006-05-29T16:26:00.000+01:00

NNnit is a great tool.

However, it has one deficiency. As far as I can tell, you cannot easily write data driven tests that appear in the results.

If I have a file, that contains test a value and a result, I can read the file line by line (or use xml). I can then test function f

The test is then


< Test() > _
Sub Test
    for each line
        assert.IsEqual(line.result, f(line.value))
    next line
End Sub

The problem is that each line will not be tested. The test fails on the first failure.

At the same time, I only see one result in the output - test.

It would be better to have all lines appear as tests in the output.

Perhaps something like this.


< Dynamictest() > _
Sub Test()
    for each line
        StartTest("f(" + line.value + ")"
        assert.IsEqual(line.result, f(line.value))
        EndTest
    next line
End Sub

Effective Date and Notice Date

2006-05-27T10:59:00.000+01:00

If we consider an accounts package, it is very common for details of transactions, entries, to be entered after they have taken place. There are two dates that need to be recorded. First is the date and time the transaction is entered. Second there is the effective date, or the date on which the transaction happened.

Given these dates there are some quite complex queries that can be written that answer some difficult questions.

For example, we can ask the question, what were the accounts at the end of January?

Select * from Entries where notice_date <= #31-JAN-2006#

We can also ask what should the accounts have been at the end of January?

Select * from Entries where effective_date <= #31-JAN-2006#

Very subtle difference between 'were' and 'should'.

There is another feature of accounts. Accountants like to close the accounts for a period. That means they don't want to accept any new entries where the effective date is earlier than the period end.

So lets say we want to close the books for the end of the year. We allow entries to be made up to the end of January, but after that we will close the books.

The first question will be what are the accounts at the end of the year.

Select * from Entries where effective_date <= #31-DEC-2006#

All very simple.

It is also important to know what adjustments were made after year end.

Select * from Entries where effective_date <= #31-DEC-2006# and notice_date > #31-DEC-2006#

All simple.

There are two approaches to locking the accounts down. One is to not allow any entries made after the end of January with an effective date in the last year. That is no back valued transactions that affect the previous end of year accounts. The alternative is to allow them, and change the queries.

The end of year accounts then become.

Select * from Entries where effective_date <= #31-DEC-2006# and notice_date <= #31-JAN-2006#

Here then 31-JAN-2006 is the closing date of the books.

You can continue making adjustments to the previous year's books. However, the entry screen should warn that this is happening.

We then add one more query. Next closing of the books, we need to know the adjustments to the previous year's accounts.

Select * from Entries where effective_date <= #31-DEC-2007# and notice_date > #31-JAN-2006#

and the actual accounts themselves uses the same query

Select * from Entries where effective_date <= #31-DEC-2007#

Just with the addition of two dates, we can get queries that distinguish between two important questions, what should have been the state and what was the state.

Time times for commuting

2006-05-26T14:48:00.000+01:00

Travel-time Maps and their Uses: "Transport maps and timetables help people work out how to get from A to B using buses, trains and other forms of public transport. But what if you don't yet know what journey you want to make? How can maps help then?
This may seem a strange question to ask, but it is one we all face in several situations:
Where would I like to work?
Where would I like to live?
Where would I like to go on holiday?
These are much more complicated questions than those about individual journeys, but one thing they all have in common is transport: can I get to and from the places I'm considering quickly and easily?
The maps on this page show one way of answering that question. Using colours and contour lines they show how long it takes to travel between one particular place and every other place in the area, using public transport. They also show the areas from which no such journey is possible, because the services are not good enough."

A great idea that I've been thinking about for a while. All they need to do is turn it into a production system.

What I couldn't work out was an easy way to get the travel time. What this group has done is interesting. They have used a proxy for the travel time. They picked the time taken to arrive at a place at 9.00 am. In other words, the time most likely to be used by people when using the map in the first place.

What is needed is a website for this to be built where you can input details of your journeys. Reading the article it looks like the Ordnance Survey is fulfilling its government brief of making money out of the taxpayer for something that the taxpayer has already paid for. All government information should be free.

One extension I can think of is this. You might have a set of transport needs. My commute to work, visiting my parents, travelling to my favourite shops and restaurants, playing sport. I should be able to set up these journeys and give them a weight. Now the isochrones for each journey can be calculated, multiplied by the weighting and totaled. That then gives me the locations that minimise or optimise my transport needs.

Bidirectional Debugger

2006-05-25T11:52:00.000+01:00

Undo Software announces UndoDB: "A bidirectional debugger allows programmers to run a program backwards in time as well as forwards. The program can be stepped back line-by-line, or rewound to any point in its history. Furthermore, programmers can play the program forwards and backwards in a totally repeatable fashion, allowing them to 'home in' on the cause of a bug.
Bidirectional debuggers are much more powerful than their traditional counterparts, which only allow programmers to step their programs forwards in time. This is particularly true for bugs whose root cause occurs long before the ill effects manifest themselves, and for bugs that occur only intermittently. "

This is an interesting idea. Debugging both ways!

"With this analogy, a programmer using a bidirectional debugger is like the detective finding detailed CCTV footage not just of the murder itself, but also all pertinent events that led to the murder."

Martin Fowler - Code Ownership

2006-05-13T10:19:00.000+01:00

Martin Fowler - Code Ownership: "There are various schemes of Code Ownership that I've come across. I put them into three broad categories:
Strong code ownership breaks a code base up into modules (classes, functions, files) and assigns each module to one developer. Developers are only allowed to make changes to modules they own. If they need a change made to someone else's module they need to talk to the module owner and get them to make the change. You can accelerate this process by writing a patch for the other module and sending that to the module owner.
Weak code ownership is similar in that modules are assigned to owners, but different in that developers are allowed to change modules owned by other people. Module owners are expected to take responsibility for the modules they own and keep an eye on changes made by other people. If you want to make a substantial change to someone else's module it's polite to talk it over with the module owner first.
Collective code ownership abandons any notion of individual ownership of modules. The code base is owned by the entire team and anyone may make changes anywhere. You can consider this as no code ownership, but it's advocate prefer the emphasis on the notion of ownership by a team as opposed to an individual."

This is an interesting observation. Psychologically, people like strong code ownership. If you take a pride in your work, it is almost inevitable.

I have worked on weak code ownership as a system. Forced integrations every two weeks. Negotiations with the other owners for changes to their code. Integrator is responsible for resolving other issues.

However, as you move towards continuous integration, I think there is more justification for collective ownership. After all, you still get the psychological satisfaction of knowing who broke the code! :-)

Simian

2006-04-30T17:22:00.000+01:00

Simian is a simularity analyser. It looks at your code and finds similar code else where. Ideal to get a grip on cut and paste coders.

This is really a tool to deal with the problem that you should only code your program once. Just as with a database, only store your data in one place and don't duplicate the same approach should be taken to coding. This is the principle behind the 2nd normal form for a relational database.

I can think of one exception where Simian would give an incorrect result, and that is with code generation. Code generation is likely to produce lots of duplicate code, but it doesn't break the principle that you shouldn't duplicate code. The reason is that it is the compressed representation of the code, and the code generator itself that is important, not the output.

Well almost. If you do have lots of duplicate code, then you are going to have a larger program than necessary. Simian can help detect this, and you can then feed this back into your code generator, to optimise the output.

FxCop and Nunit

2006-04-28T12:50:00.000+01:00

PMD has a set of JUnit rules. It would be interesting to implement them in FxCop

JUnitStaticSuite: The suite() method in a JUnit test needs to be both public and static.
JUnitSpelling: Some JUnit framework methods are easy to misspell.
JUnitAssertionsShouldIncludeMessage: JUnit assertions should include a message - i.e., use the three argument version of assertEquals(), not the two argument version.
JUnitTestsShouldIncludeAssert: JUnit tests should include at least one assertion. This makes the tests more robust, and using assert with messages provide the developer a clearer idea of what the test does.
TestClassWithoutTestCases: Test classes end with the suffix Test. Having a non-test class with that name is not a good practice, since most people will assume it is a test case. Test classes have test methods named testXXX.
UnnecessaryBooleanAssertion: A JUnit test assertion with a boolean literal is unnecessary since it always will eval to the same thing. Consider using flow control (in case of assertTrue(false) or similar) or simply removing statements like assertTrue(true) and assertFalse(false). If you just want a test to halt, use the fail method.
UseAssertEqualsInsteadOfAssertTrue: This rule detects JUnit assertions in object equality. These assertions should be made by more specific methods, like assertEquals.
UseAssertSameInsteadOfAssertTrue: This rule detects JUnit assertions in object references equality. These assertions should be made by more specific methods, like assertSame, assertNotSame.
UseAssertNullInsteadOfAssertTrue: This rule detects JUnit assertions in object references equality. These assertions should be made by more specific methods, like assertNull, assertNotNull.
SimplifyBooleanAssertion: Avoid negation in an assertTrue or assertFalse test. For example, rephrase: assertTrue(!expr); as: assertFalse(expr);

Code Analysis Tools

2006-04-28T10:09:00.000+01:00

I am a heavy user of Fxcop. It is a code analysis tool that works on DotNet assemblies. That means it works with multiple languages, since it is operating at the compiled code level.

It is quite picky, but it is possible and worthwhile getting zero errors.

However, there are some errors that it cannot catch. These relate to information that is lost when you compile. I've come across two examples.

First is imports (or using). It is very easy to get extraneous imports. The compiler removes them, and so FxCop cannot tell if they exist in the source code.

Secondly relates to comments. There are great applications like NDoc. NDoc takes code, strips out XML comments and turns into documentation. Javadoc is another example. However, it is possible to change the code, and not keep the documentation up to date. For example, there is no check the signature of the function matches the signature in the documentation. FxCop like tools cannot check, because the comments are thrown away.

Interestingly tools for Java like IntelliJ handle this. You need to search through the code and find out exactly which imports are used, and could they be simplified.

Should be part of any good IDE.

The Problem with Unit Testing

2006-04-26T23:19:00.000+01:00

One issue with unit tests is the limited number of paths through the software the tests test. Secondly, I'm not convinced that writing the test first, is the best way of writing code.

Taking each of these points in turn. You can address part of the limit number of paths by combining a code coverage tool with your unit tests. If you don't get code coverage, you can add more tests. However, what you really need is something that tests all the time.

Secondly, writing the test first focuses on a particular case, not on producing a good abstraction or good code. If you are not careful, the code is produced to meet a particular test case, not the set of test cases, or even the set of possible data.I've worked for a long time with Eiffel, and it has the interesting feature of design by contract. This combined with unit tests takes things to the next level. A lot of people who comment on DBC, have never used DBC in practice.

First, you change the way you program. It isn't write unit test first, it is contract first. Then write code / unit test. The change of emphasis is produce code that covers all cases. All cases are defined by the contract. Now it becomes clearer what the test cases are. You need to exercise the cases defined by the contract. Whether you write the tests, or the code first doesn't matter. Importantly, if you run the code with assertion checking on, you are testing the contracts as you go. Instead of testing just specific cases, you get universal testing. It is far more powerful.

There are some other alternatives that I can see coming in the future. FxCop and other static code analyzers are getting more sophisticated. Software such as the proposed NStatic are also interesting. Combining these with DBC would make something really powerful. I can also see the day when run time analysis during testing feeds back into the static analysis

Welcome

2006-04-15T17:12:00.000+01:00

Welcome - This is a blog about developing software.