Writing unit tests can easily become a false sense of security; “all my tests pass, so surely my application must work.” But, how do you know for sure that you’ve tested everything? Using TestDriven.NET, I can run all my tests with code coverage reporting through its integration with NCover, by right-clicking on the solution folder which holds all my test projects and selecting ‘Test With - Coverage’:

and opens up NCover Explorer with the coverage report:

This is interesting; from the results we see here, it looks as if my test coverage is far from perfect. Only 58% for my backend code? Yikes! But if we drill into the assembly, we get a different story:

Looking at this, I know that the Properties namespace is just some Visual Studio generated code – I don’t care about testing that. Additionally, the Model namespace has 55% coverage – which again sounds just about right, because most of the code in this namespace is code auto-generated by the Linq to Sql designer anyways. So if I exclude those two namespaces from the results (hit Del), the overall coverage for my backend assembly suddenly jumps to 100%.

The moral here is that we usually need more knowledge about the code base than simply looking at the code coverage results can give us in order to really determine what parts of the code are thoroughly tested and which parts are not. To some extent, we can use our own knowledge about the code to draw conclusions about the results, like I did above. However, relying on our intuition about which parts of the code are ‘important’ can be a bit dangerous – what do we really know, anyway? After all, programmers are notorious for underestimating.

Augmenting Test Coverage with Static Code Analysis

What we need, is more tangible evidence about the state of our codebase, and performing some form of static code analysis can give us just that. Visual Studio has some static code analysis functionality built in, but for this post I’ll be using the much more advanced tool NDepend for my examples.

A popular static code analysis metric you may have heard about, is cyclomatic complecity. It tells us something about the complexity of the method by analysing the number of execution branches that exists within it: Long methods with lots of conditional statements will have a high complexity, a sign that the method is hard to understand and may be susceptible to bugs – and more importantly a sign that such a method needs to be tested thoroughly (if not refactored into something more manageable, that is). There are lots of other metrics as well – if you want to learn more about static code analysis I recommend you check out the Metrics Definitions page over at the NDepend site, or read this great introductory article by Scott Hanselman.

One of the coolest features of NDepend, is that we can import the test coverage report from NCover 2.x (or Visual Studio), by going to the Analysis page in Project Properties:

This gives us the power to combine what we know about the source code through the static code analysis, and augment it with the test coverage analysis.

NDepend has a feature called CQL, Code Query Language, which allows us to write queries using SQL-alike syntax in order to find out things about our code. After importing the code coverage results, we can now for example query the code base for any method that is fairly complex and which does not have 100% test coverage:

SELECT METHODS WHERE PercentageCoverage < 100 AND CyclomaticComplexity > 10

This is a naive query though - a better one is the default “Complex methods should be 100% covered by tests” constraint that ships with NDepend, which looks at a set of relevant metrics:

WARN IF Count > 0 IN SELECT TOP 10 METHODS  WHERE 

    (  NbILInstructions > 200 OR 

        ILCyclomaticComplexity > 50 OR 

        ILNestingDepth > 4 OR

        NbParameters > 5 OR 

        NbVariables > 8 OR

        NbOverloads > 6 ) AND 

    PercentageCoverage < 100

There are lots of ways we can combine the metrics to come up with interesting results – we could for instance look at which types and methods are the most used in our application (using metrics like method rank and afferent coupling), and ensure that we cover these with lots of tests. Patrick Smacchia has a great post where he goes more in depth on how to take advantage of the combination of static code and code coverage analysis.

Running these queries on my code base, I find very few methods that seem to need more thorough testing. How confident should that make me that my code is well tested and does not contain any major bugs?

100% Coverage, a False Sense of Security?

There are lots of ways to calculate test coverage, and the naive statement coverage method that most tools use by default has several disadvantages – most notably that code often has many decision points and loops, and simply measuring that all statements have been executed can easily yield false negatives. Branch coverage on the other hand, measures which of the possible execution paths through the code have been exercised. Take the following method, for instance:

public string Silly(bool someCondition)

{

    object val = null;

    if(someCondition) val = new object();

    return val.ToString();

}

With the following test, statement coverage will report 100% for this method:

[Fact]

public void Silly_returns_a_string_when_condition_true()

{

    Assert.NotNull(Silly(true));

}

But if someone calls Silly(false), then clearly it will blow up! Branch coverage, however, would report < 100% for this method, because the branch on which someCondition = false was never executed by the tests. Both NCover and NDepend support branch coverage metrics, so we could for instance run the following query to find all methods like the above:

SELECT METHODS WHERE PercentageBranchCoverage < 100 AND PercentageCoverage == 100

If you want to know more about code coverage analysis and the different ways of calculating it, then have a look at Steve Cornett’s paper.

Knowledge is Power

In the end, the more you know about your code the better equipped you are to make sure it will do what it’s supposed to. Using tools like NDepend for static code analysis and NCover for code coverage analysis can be of great help in not only gaining more insight about your code, but also guiding you to the parts of the code which are most important to the application as a whole.

I’m going to keep this weeks ReSharper tips really simple, as the weather is way too nice to be stuck indoors - at least here in Norway; the sun is shining from a cloudless sky, seagulls are cawing in the distance, and temperatures are above 20 degrees even this late in the evening (and I hear talk that the water temperature is not too bad out in the fjord either, although I might just wait a bit longer before diving in… :p).

But I digress... Today, we’ll look at possibly one of the most finger-twisting ReSharper keyboard shortcuts: Ctrl+Shift+Alt, [Direction Keys]. Yeah, not exactly easy on the fingers that one – but it’s a really handy feature.

By using the Code Clean-up feature (Ctrl+E, Ctrl+C), you can have ReSharper order all the members in your classes alphabetically. However, sometimes that’s not quite what you want. Instead, placing your cursor on for example the line of a method signature, you can press Ctrl+Shift+Alt and then use the Up and Down arrow keys to sort it manually. When you invoke the key combination, the block you’re moving is coloured in blue:

You can move just about anything this way, not just members in a class – statements within a method works too, for example:

You can move things left and right too, not just up and down. For instance, you can swap the left and right-hand side of an expression:

…or reorder the parameters of a signature:

Note however that reordering parameters this way will not safely refactor any usages of this method - for that, you should instead use Ctrl+R, Ctrl+S which brings up the Change Signature dialogue.

And that’s about it really – a simple, handy feature that will help you keep your code files neatly organized.

Previous ReSharper Tips of The Week

#1 – Finding Things
#2 - Navigation

Last week I started a new series of posts, in which I'll be presenting some of my favorite ReSharper shortcuts. Last time we looked at finding things; this time we'll focus on navigating. Note that I'm using the Visual Studio ReSharper scheme (which is the default); if you're using another setup then the shortcuts will differ. You can change this from the General pane of the ReSharper options dialogue.

Navigating Inheritance

When the cursor is on a type name, hitting Ctrl + End brings up the 'Derived Types' dialogue. From here, you can quickly navigate to any type that inherits the type you invoked it for:

Similarly, Ctrl + Home brings up the 'Base Types' dialogue, which allows you to quickly navigate in the other direction:

If there is only one derived type or one base type, then ReSharper will immediately navigate to it, skipping the dialogue (at least, this is the default behaviour - you can change it from the configuration if you want to). Note that the cursor doesn't have to be on a type declaration for this to work, it works anywhere a type name is found (such as in variable declarations, for instance).

Navigating Member Inheritance

If the cursor is on a member name, then Ctrl + End and Ctrl + Home will instead let you navigate to types which override it, or base types from which it was overridden, respectively:

For member declarations, ReSharper also provides a nice visual hint that the member implements an interface or overrides a base member, in the gutter of the editor:

Here for example, it tells us that the property is an implementation of an interface; notice the green "i" icon (circled in red). If you click the icon, ReSharper will take you to the interface where the property is declared. If the declaration is in a type outside your solution, ReSharper will bring it up in the Object Browser instead (this goes for all navigation shortcuts, incidentally).

Navigating to Usages

If the cursor is on a member name, hitting Shift + Alt + F12 will bring up a dialogue showing all the usages of this member:

Naturally, this list can be quite long. Hitting Shift + Alt + F11 will instead only search for usages within the current file, and instead of showing a list it will highlight the declaration in red and the usages in blue:

The usages remain highlighted until you press Esc.

Navigating to Declarations

If the cursor is on a member name, hitting Ctrl + Shift + F11 will take you to the type that declares this member. If the cursor is on the instance of something, this will take you to the type declaration of this instance.

Navigate from Here to...

Memorizing all these different shortcuts can be a bit daunting to begin with, and its easy to just revert to using the mouse. Don't despair just yet though; all you really need to begin with is to remember Alt + \ (that's a backlash), which will show you what navigation options are available in the current context:

Notice that it also shows the direct shortcut next to each option, which will help you learn them in no time.

I caught wind of this from Rune Grothaug's blog; he writes that Microsoft Research has published the beta of a Visual Studio plugin called PEX - Program EXploration. Basically, it is a tool that generates unit tests. It works by having you write what they call a "parameterized unit test", from which PEX then helps you generate a traditional test suite through an iterative process. It looks pretty awesome, and seems to be very test-first oriented as well, which is great. Below is a grab from the screen cast they've published showing it in action. Go check it out here - I'm definitively going to download this and have a play around with it sometime this weekend :)

Build 804 of ReSharper 4 has been tagged as a "beta candidate, stable". Does this the mean a full release is just around the corner? I sure hope so! I've been using the nightly EAP builds since mid-February, and although they've mostly been working OK, there's certainly been a fair share of bugs and poor performance. To be expected from unreleased bits, of course, but very frustrating still.

Anyway's, I have a feeling there are a lot of ReSharper users out there these days - I've at least met a lot of people who use it (we actually had trouble finding someone to give away a free license of it to at the last local .NET user group meeting, as almost everyone already had one!), so I thought I would start a series of posts which highlight some of its features I find the most useful. I'll try and post a new tip at the start of every week, until I run dry (or get bored doing it :p).

To start things off, we'll have a look at finding things (not car keys, though; sadly ReSharper can't help out there):

Finding Types

Hit Ctrl + T anywhere, and you'll see the "Find type" dialogue. This allows you to search for and jump to any type declaration within the current solution:

It also supports filtering by CamelHumps, which allows you to skip lower-case letters in the search string. Notice how it finds the types with a first word that starts with "D" and a second word with starts with "Cont":

You can also use wildcards:

Finding Other Things

As well as the ability to find types, there's an equivalent dialogue for finding files, which is accessed by hitting Ctrl + Shift + T. It will find any type of file within the solution, with the same features:

And finally, there's the Find Symbol dialogue, which searches for - yes, you guessed it - symbols. It is activated by Alt + Shift + T, and will find any symbol of any access in any type declared within the current solution:

That's it for part one. Next time, we'll look at more ways to quickly navigate your codebase.

For a while, I've been using Color Blender to get inspiration for color themes when designing web sites. However, Adobe's first web-hosted application Kuler might just be even cooler (heh, heh).