I have been looking at mpstat output for some time now and I noticed that none of my LPARs were showing s4rd thread migrations, even though the lssrad -av output showed than the LPAR was spread over more than one Chip in a Node.  I mistakenly believed that all my threads were being re-dispatch back to the cores within the same Chip as they were last running and that AIX was dealing with the workload efficiently.

But then I noticed on a test LPAR that it was actually reporting s5rd thread migrations even though the lssrad output showed all the virtual CPUs were contained within a single Node and this was confirmed with PowerVP on my E880.  At the time of my testing, there were no other workloads running on the E880 (except the VIOS) so I was fairly confident that the data I was seeing was correct.

Please note that in this configuration, the S3hrd and S4hrd thread memory accesses are being reported correctly.

After raising a PMR with IBM, it has finally been confirmed that there is a bug in the kernel dispatch code that is mistakenly incrementing the S5rd counters when infact it should be incrementing the S4rd counters.

As draft APAR has been opened and is awaitng a fix (as of early November 2016).

IV92886 => AIX 7.2

IV96814 => AIX 7.1

IBM have told me that both fixes will be available sometime in Q3 2017 in the following releases.

AIX 7.2 TL2 base

AIX 7.1 TL5 base

 

Here is the lssrad -av output.

REF1  SRAD  MEM       CPU
0
         0  15658.31  0-3 16-19 32-35 48-51 64-67 80-83 96-99 112-115 128-131
         1  15687.00  4-7 20-23 36-39 52-55 68-71 84-87 100-103 116-119 132-135
         2  15687.00  8-11 24-27 40-43 56-59 72-75 88-91 104-107 120-123 136-139
         3  15900.94  12-15 28-31 44-47 60-63 76-79 92-95 108-111 124-127 140-143

 

Here is a sample of the mpstat output.

The full output of the mpstat data can be found in the attached text file.

System configuration: lcpu=144 ent=36.0 mode=Uncapped

cpu   cs  ics bound   rq push S3pull S3grd  S0rd S1rd S2rd S3rd S4rd S5rd ilcs  vlcs  S3hrd S4hrd S5hrd %nsp

ALL 1111 1011   102  102    0      0   504  87.4  3.6  0.0  0.5  0.0  8.5    4  2698  95.0    5.0   0.0    0