This is a series on my findings around building open source software online for free. There are many SAAS (Software As A Service) platforms out there and a lot of these are free to use with your open source projects. But not all of these work with C# and .Net. So this series highlights the tools and products out there that work with .Net in different phases of the software development life cycle.
- Build
- Test <=(We are here)
- Publish & Deploy
- Documentation
- Metrics
- Lifecycle Management
Test
Once we Have our Build Server and Build Scripts all chosen and set up, we can add value by running our tests to check the integrity of our code. Once we have these tests running every build for every check-in, we can monitor the health of our code base, of our features and branches. There are plenty of options to choose from for test frameworks so lets have a quick look at a few specifically.
- MSTest
- NUnit
- xUnit
- Fixie
- SpecFlow
And as a secondary to test, usually we execute our test runs in the context of a coverage application, so we can capture and report on the metrics from how much of our code is covered by our tests. There is plenty of discussion about the value of this, but most people would agree that knowing you have low coverage and that encouraging the writing of more tests is better than not knowing your coverage at all.
- OpenCover
- SharpCover
- monocov
- XR.Mono.Cover
- Visual Studio
MSTest
MSTest is the de facto, ships with Visual Studio out-of-the-box test framework most people use. It has the built in templates, a basic set of features that gets you most of the way there, and just works. That is until you move to mono. The is a slight coupling to visual studio if you decide to base your tests on this test framework, which means if you want to go cross platform you need to rewrite all your tests. Maybe a good starting point, but you will eventually want to move off this either for compatibility or features. Having said that, you will still get a long way with MSTest.
NUnit
NUnit is a popular open source alternative to MSTest. The framework takes the basic principles of MSTest and extends them in a few ways to enhance the flexibility of its use. It basically gives you the ultimate in flexibility in how you wish to organise your tests, including parameterised test case attribution to your tests, all the setup and teardown options and all the runners and integration to tools you need. Version 3.0 (a complete rewrite) is currently in alpha.
xUnit
Another slightly less popular but rising test framework is xUnit. This one is a bit more opinionated, and tries to cut the scope of available features compared to NUnit and MSTest around Setup and Teardown, opting for constructor and destructor as a more native approach. However it replaces per method Setup and Teardown with a slightly different dependency model using Fixtures (implementing IUseFixture
Fixie
Fixie is something I’ve only come across in passing. It looks to be a very promising test framework, and I encourage you to take it for a test drive if your looking for something to try. My understanding is it is the bare minimum scaffolding to run tests. You are then left to write your conventions for defining and finding what is a test, and defining your own assertion library (or just pick one from NuGet). It allows you to write your own expressions to wrap test methods with classes (for Setup and Teardown) and defining your own parameter injection resolution.
If you find your highly opinionated in how you want your tests to work, and don’t quite like some small parts of your chosen framework, this might be the one to use to get things exactly right.
SpecFlow
SpecFlow is a Specification by Example, Behaviour driven development style test framework. It uses natural language text files to produce and consume methods containing partial test steps that a developer can implement. this approach means that someone can write up the acceptances tests as tests and not have to write code. They can even compose the existing methods from previous tests into new acceptance tests my chaining them together in new ways using the Gherkin DSL. If you buy into this approach then it might be worth giving this one a go. Usually these test are used with User Automation Testing, where you can describe use actions in plain English and be able to test this behaviour.
OpenCover
This appears to be the best open source coverage tool available. OpenCover works with most if not all of the test frameworks, and even has a OpenCover.UI extension for visual studio being actively worked on. The only downside I have found is it uses C++ COM to communicate with .Net, and so does not work with mono.
SharpCover
SharpCover is another alternative open source coverage tool. This one is built on mono.cecil and works cross platform. I am yet to actually try this one out, so cannot give much feedback other than that it is cross platform, and hasn’t had a release in over a year. But I did get a quick response from the issue I raise recently.
XR.Mono.Cover
XR.Mono.Cover looks really promising. This appears to be a mono-only coverage, but claims to work on mono on windows as well. Again I have no experience to share here. It does have a GUI component and stores its data in a SQLite database for easier data analysis later on. It is a bit light on updates, so either it doesn’t have any issues, or no ones really using it.
monocov
Monocov is an open source coverage tool for mono. I managed to get this working despite it being several years unmaintained, and it is pretty broken. If you really want to do coverage on mono I would look into XR.Mono.Cover and SharpCover.
Visual Studio
Visual Studio has had coverage capabilities is some of its higher SKUs for a while. And as well as the GUI coverage you can use in the test runner, it also has command line tooling you can use. you can even produce a coverage report using a flag on the vstest.console runner. Not really open source, but a nice option if you have it available.
And More
As usual this is just a cross section of Frameworks and coverage out there. For a reference list there is the List of unit testing frameworks, .Net section on Wikipedia that lists a few, as well as the various runners available. Before picking the one to go with, I suggest verifying it fits into your preferred way of running tests. Most have a pretty good command-line option, integration points into CI Servers, and the Resharper and Visual Studio test runners. it is worth checking out dotcover and NCover because they have available free licensing for open source projects.
And as one last optional extra, there is a tool to generate coverage reports from most of the coverage tools I mentioned above called ReportGenerator.
Next up is how we publish and deploy our newly built and tested applications!