Interrupt affinity and Microbenchmarks

One of the purposes of a micro-benchmark such as NetPerf in a development environment is to track performance over time in a very deterministic manner. This is increasingly at odds with the non-determinism caused by interrupt and application scheduling over large numbers of CPU cores. For this reason, we often pin both the interrupt and application when running a single-stream micro-benchmark.

To pin interrupts to a given CPU core using Linux you:

- stop the irqbalance service
- cat /proc/interrupts to determine which MSI-X interrupts (N) are allocated to your device
- echo (1<< CPU) > /proc/irq/<N>/smpaffinity

To pin an application to a given CPU core using Linux you:

- taskset (1<< CPU)  <application command line>

These days, similar facilities are available for all other OS, but conversely for multi-stream testing where we want to measure the effect of receive side scaling techniques interrupt and application pinning is generally counterproductive.  However it's always an amusing exercise to see whether you can manually do a better job than the OS/hypervisor for a given workload.