I have been looking at mpstat outputs for some time now and I have raised a number of PMR calls with IBM as I have found several discrepancies in the outputs.  I have already posted one blog about mpstat incorrectly reporting thread redispatch counters and also discovered that the S5hrd counters can be incremented incorrectly when you have memory only SRADs sharing the same REF id.

I am now sharing my understanding of the mpstat output regarding S0rd, S1rd, S2rd, S3rd, S4rd, S5rd thread migrations and the S3hrd, S4hrd, S5hrd memory affinity counters as I believe many people misunderstand this data and what exactly it is telling you.  Please note this is my interpretation of the data based on PMR questions and answers with AIX kernel developers and Performance Specialists and results seen during testing in my p8 environment.

 

Before I continue, we should first review the lssrad -av output (see post here) as the counters that mpstat reports are based on AIX's view of the LPAR.

Yes, you read that correctly, the output of mpstat is based on AIX's view of the LPAR as returned by lssrad -av.  This means that if you have an LPAR that has been told it is contained within a single Node and Chip, it should never report S4rd or S5rd counts, nor will it report S4hrd or S5hrd counts.  To confirm this I asked the following question to IBM Performance Support.

 

Q1 = So the mpstat statistics for S?rd and S?hrd are based on AIX's view of the hardware reported by lssrad output, regardless of where the VPs are actually dispatched by the Hypervisor.

A1 = Those "redispatch" statistics(S*rd) are maintained relative to the last logical CPU that a thread was dispatched on.  And, the home SRAD redispatch statistics(S3hrd, S4hrd, S5hrd) are maintained based on logical CPU SRAD affinity on AIX.  So, it may be a little tricky in a SPLPAR (Shared Processor Pool) environment because the physical processor mapping of the VP logical CPU SRAD we see from AIX can change dynamically according to shared pool usage.

 

This was an interesting answer and I confirmed the response with the following questions, to which the answer was: "Basically,  the answer to all your questions is yes".

 

Q2 = So if my LPAR has many VPs (more than a single Chip), but reports the lssrad output like this, I will never see S4rd or S5rd as AIX believes the LPAR is contained within a single chip, even though the Hypervisor could schedule VPs to local or remote chips?

# lssrad -av
REF1  SRAD      MEM       CPU
0
         0     7662.69    0-47

 

Q3 - If my LPAR reports an lssrad like this, I may see S4rd and/or S4hrd due to local thread scheduling within AIX and local memory access.
# lssrad -av
REF1  SRAD      MEM       CPU
0
         0    19456.00    0-11 16-19 28-31 40-43 52-55 64-67 76-79
                          88-91 100-103 112-115 124-127
         1    13145.38    12-15 24-27 36-39 48-51 60-63 72-75 84-87
                          96-99 108-111 120-123
         2    14820.00    20-23 32-35 44-47 56-59 68-71 80-83 92-95
                          104-107 116-119


Q4 - If my LPAR reports an lssrad like this, I could see S4rd + S5rd and S4hrd + S5hrd due to local and far thread scheduling and local and
far memory access.
# lssrad -av
REF1  SRAD      MEM       CPU
0
         0    32749.12    0-63
         1     9462.00    64-67 72-75 80-83 88-91
1
         2     2471.19    92-95
2
         3     1992.00
         4      249.00

 

Which LPAR configurations are most affected?

Dedicated LPARs.

LPARs that are running with dedicated cores are always scheduled back to their home cores by the Hypervisor.
Therefore the thread redispatch statistics from mpstat should be a true representation of the redispatches to the physical cores.

 

High Entitlement LPARs.

LPARs with a very high entitlement to Virtual Processor ratio are very likely to be scheduled back to their home cores if they are under entitlement.  Therefore the thread redispatch statistics from mpstat should be very close to a true representation of the redispatches to the physical cores.

 

Low Entitlement LPARs.

LPARs with a low entitlement to Virtual Processor ratio are not guaranteed to be scheduled back to their home cores if they are running over entitlments.  This is especially true if there are other LPARs that are homed to the same cores that are running under entitlement, as these LPARs will get the home cores first.  Therefore the thread redispatch statistics from mpstat may not show a true representation of the redispatches to the physical cores as LPARs running over entitlement could be schedule to any free cores within the frame.