Friday Links 0.0.3: BenchmarkDotNet

This is based on an email I send my .NET team at work

Happy Friday,

BenchmarkDotNet

https://github.com/PerfDotNet/BenchmarkDotNet

I’m starting to see this tool become the de-facto standard for writing C# benchmarks.

It papers over a lot of the things about benchmarking .NET code that can be tricky: like handling garbage collection, and warming up the Just In Time compiler (JIT).

To use it, you just make a class and decorate a couple methods with the [Benchmark] attribute and call into the BenchmarkRunner from your console app’s Main method.

It dumps logs in the bin directory and prints useful information to the console.

You can also configure extra diagnostics like GC times and memory allocation.

If you have some time this afternoon, I’d like to see some benchmarks. Try creating a test of a couple different ways of doing something. For example, find out how much slower LINQ to objects is over a hand-rolled for loop.

Most interesting benchmark gets bragging rights.

Thoughts on Benchmarking

There are a lot of traps in writing good benchmarks. For a fair comparison you want to run a RELEASE build. In RELEASE, the compiler does optimizations that can make an enormous difference in run-time, for example method inlining or loop unrolling. It doesn’t optimize in DEBUG so that when you are stepping through in the debugger, the code being analyzed actually matches the source code you’re looking at.

I mentioned earlier that in .NET especially, you have to be aware of Garbage Collection and JIT overhead. A good benchmark will run GC cycles before each test to make sure each run is as clean of an environment as possible. You also need to call the methods under test once so that the JIT compiler has a chance to convert the MSIL byte code into native machine code.

The last thing I’ll say about benchmarking is to beware of is the pull towards micro-optimizations or spending too much time optimizing code paths that don’t matter. It’s well known, for example, that LINQ is slower than hand-rolled for loops. But on the other side of a 50ms DB query, does it really make any noticeable difference? Unless your project is running at an extreme scale, there are much better areas to spend your optimization time: tuning database queries and minimizing external API calls are usually where you’ll get the most bang for your buck.

That’s not to say there is no customer value in an optimized system, or that you shouldn’t give any thought at all to performance. But its often cheaper for clients to throw money at better hardware than to pay for 8 hours of your time hunting down another 3% perf gain.

Make good algorithm choices, and then do profiling to learn where the real hot spots are.

P.S. this is frightening:

From the docs
string url = "<E.g. direct link to raw content of a gist>";
var summary = BenchmarkRunner.RunUrl(url);