Starting with the VIO Server version 2.2.4.0, the concurrency level controls the number of threads that are used to send and receive the memory pages of the mobile partition.  More threads require more processor and network bandwidth and can significantly improve the LPM data copy times.

In this post, we look at the LPM data copy times for at 64GB, 128GB, 256GB and 512GB LPARs with the different LPM concurrency levels.  For these tests, we used the PCIe3 40GbE RoCE Converged Adapter and the LPMs were between an S824 and an E880.  No other workload was running on these frames at the time of the testing. 

Please refer to this IBM link that details the recommended network bandwith required for each concurrency level.  IBM Link for Concurrency Levels

The current concurrency level can be displayed with:  lsdev -dev vioslpm0 -attr concurrency_lvl

To ensure that the LPM copied all of the LPARs memory, you have to ensure that all memory pages have been touched.  Any memory pages in an LPAR that have not been written to will not be copied over the network.  To ensure all the memory pages were touched, I created RAM disks for about 90% of the LPARs memory.  I also had a script that copied a 1GB file to those RAM disks while the LPM was in progress to simulate IO load on the LPARs.

Each of my VIO Servers were running VIOS 2.2.4.21 with 4 Dedicated CPUs and 12GB of memory.

Total LPM runtime in seconds is on the left and LPM memory copy time is on the right.