Power System Hypervisor Resource Dumps
What are Hypervisor Resource Dumps?
Hypervisor resource dumps are text based reports from the Hypervisor partition in the Power System that give you a detailed report on the Hypervisors view of your LPARs configurations, along with details on activated CPU and memory resources. These reports are in sections, detailing Frame configurations, LPAR overviews, Resource Licensing, Shared Processor Pools, Partition CPU and memory settings, and Domain Affinity Reports. There is a final detailed Affinity Report that contains DPO affinity scores and affinity group data.
How to Start a Hypervisor Resource Dump
According to the IBM tech note, it is safe to run a resource dump but this only applies to FW350, FW760, and later Firmware levels.
http://www-01.ibm.com/support/docview.wss?uid=nas8N1021025
Login to your HMC as hscroot and run the following command.
# startdump -m Server-name -t resource -r "hvlpconfigdata -base -procs -memory -domain -affinity"
Wait about 10 seconds, then check the /dump directory for a new Resource Dump file.
/dump/RSCDUMP.{SERIAL}.02000000.20170511001819
Hypervisor Resource Dump
Section 1 - HvLpConfigData
The HvLpConfigData report gives you a summary of your Power Server configuration, number of LPARs, Processors and memory. It also details the LMB size in MB, this is an important field to note, as the outputs later in the report refer to memory in number of LMBs.
HvLpConfigData is at 800000001538B000
..Number of LPs.............6
..Primary Lp Index..........0
..Service Lp Index..........1
..Alpha Partition Id........65535
..System is HMC Managed.....
..VIOS Allowed..............
..HMT Thread Count..........8 <== SMT Level Supported.
..System Processors Active..24 <== Active Processor Count.
..Interactive Available.....10000
..Amount of Licensed Memory.4096 <== Licensed Memory.
..Licensed Memory in Use....1813 <== Memory in Use.
..Size of HvlpConfigData... 85904
..LMB Size in MB............ 256 <== LMB Size in MB.
..Persistent Heap Storage... 113152
..Physical Real Heap Storage 517888
..Max Usable OS400 Units..*NOMAX
..Cur Usable OS400 Units..0
..Serial Port Owning Lp.....NONE (0xFFFF)
..Max Lp Id.................101
..LpMap.....................F800000000000000030000000C00007000000000000000000000000000000000
..LpMap Ids.................0 1 2 3 70 71 100 101
Section 2 - Basic LPAR Overview
The basic LPAR overview report gives you some basic LPAR configuration details, including the LPAR id, if it is the Service or Hypervisor Partition, plus the LPAR name.
The LPAR id number is the first number after the LpConfigEntryData statement. In the following three samples, the LPAR ids are 0 (Hypervisor), 1 (Service Partition) and 2 (VIO Server).
..LpConfigEntryData 0....07F000800FFF4900 Primary--Capped--Name PHYP
....Virtual Bus Pool........... 0
....Rio San Bus Pool........... 0
....Load Area Chunks........... 1
....Interative all zeros ......
....Init Weight................ 0
....Cur Weight................. 0
....Target Weight.............. 0
..LpConfigEntryData 1....07F000001526A900 Service--RPA Mode--Capped--Bootable--Name 84-SERIAL
....Virtual Bus Pool........... 0
....Rio San Bus Pool........... 0
....Load Area Chunks........... 1
....Group Id...................8001
....Interative all zeros ......
....Init Weight................ 0
....Cur Weight................. 0
....Target Weight.............. 0
..LpConfigEntryData 2....07F0000015388900 RPA Mode--Capped--Mat Shr Pool--Bootable--Hoarded--Name viost061
....Virtual Bus Pool........... 0
....Rio San Bus Pool........... 0
....Load Area Chunks........... 1
....Group Id...................8002
....Interative all zeros ......
....Init Weight................ 0
....Cur Weight................. 0
....Target Weight.............. 0
Section 3 - Licensing Information
This section of the report details your Power Systems Licensing details.
SubProcs per Core ....................... 1
Licensed Physical SubProc Count... 24
Active Licensed Physical SubProc Count... 24 <== Active Core Licensed.
Capable Physical Proc Count...... 24
Capable Physical SubProc Count... 24
Capable Logical Proc Count...... 192
Utility Procs Physical SubProc Count... 0 <== CoD Cores Available.
Active Physical Proc Count...... 24
Active Physical SubProc Count... 24
Active Logical Proc Count...... 192
Installed Physical Proc Count...... 24 <== Cores Installed in Power System.
Installed Logical Proc Count...... 192
Lp Soft Max Physical Proc Count...... 64
I5 Max Usable Proc Units.. NOMAX
Licensed Expensive Units. 2400
Available Expensive Units. 2100
Usable Proc Units by OS...
- Unknown OS Type......... 0
- AIX LPs................. 300 <== Total Entitlement for all AIX LPARs.
- Linux LPs............... 800 <== Total Entitlement for Linux LPARs.
- VIOS LPs................ 500 <== Total Entitlement for VIOS LPARs.
- IBMi.................... 0 <== Total Entitlement for IBM i LPARs.
Usable Procs by OS Type...
- Unknown OS Type......... 0
- AIX LPs................. 0
- Linux LPs............... 0
- VIOS LPs................ 5
- IBMi.................... 0
Linux Only Proc Units..... 0
Vios Only Proc Units...... 0
Section 4 - Shared Processor Pools
This section details information about Physical Shared Pools and Virtual Shared Pools. At this time, I am unsure what Physical Shared Pools are. Virtual Shared Pools are the Shared Processor Pools that you can define for your Power Systems.
Physical Shared Pools:
|------|-------+-------+-------+-------+-------|-------+-------+-------|-------|------|
|Shared| Proc Units | Pool Procs | Donor | |
|PoolId| Cur |Target |Usable |TrgtRV |CurRV |Target | Cur |Borrowd| Procs | GIQ |
|------|-------+-------+-------+-------+-------|-------+-------+-------|-------|------|
| 0 | 1200 | 1200 | 1100 | 0 | 0 | 12 | 12 | 7 | 5 | FFFE |
|------|-------+-------+-------+-------+-------|-------+-------+-------|-------|------|
These are the Shared Processor Pools defined. PoolId 0 is the Default Pool, and the other PoolId lines are any additional Shared Processing Pools (SPP) you have defined. In this example, there is 1 extra SPP with no LPARs currently assigned to it.
Virtual Shared Pools:
|------|----------|----------|----------|----------|-----------------------------------------|
| | Target | Cur | Target | Cur | |
| | Reserved | Reserved | Max | Max | |
|PoolId| Capacity | Capacity | Capacity | Capacity | Virtual Shared Pool Member Lp Indexes |
|------|----------|----------|----------|----------|-----------------------------------------|
| 0 | 65535 | 65535 | 0 | 65535 | 100 101 121 122 123 |
|------|----------|----------|----------|----------|-----------------------------------------|
| 1 | 0 | 0 | 0 | 2000 | |
|------|----------|----------|----------|----------|-----------------------------------------|
The Partition Processor Information details the minimum, desired and maximum LPAR settings for Entitlement, Virtual Processors and weight. There is also an initial (init) value that tells you how much of the resource the LPAR was given when it was first started.
Dedicated and Dedicated-Donating LPARs do not show Proc Units or Weight fields.
Partition Processor Information:
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| | Virtual | Physical | | |
| | Shared | Shared | LP | Procs / Proc Units / Weight |
| LpId | Pool Index |Pool Index|Group | | Init | Min | Max | Cur |Target |Usable |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 1 | Dedicated | 0 | 8001 | Procs | 1 | 1 | 24 | 1 | 1 | 0 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 2 | Ded-Donate | 0 | 8002 | Procs | 1 | 1 | 4 | 4 | 4 | 4 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 3 | Ded-Donate | 0 | 8003 | Procs | 1 | 1 | 4 | 1 | 1 | 1 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 70 | 0 | 0 | 8046 | Procs | 0 | 0 | 0 | 0 | 0 | 0 |
| | Uncapped | | | Proc Units | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | Weight | 128 | | | 128 | 128 | 0 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 71 | 0 | 0 | 8047 | Procs | 0 | 0 | 0 | 0 | 0 | 0 |
| | Uncapped | | | Proc Units | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | Weight | 128 | | | 128 | 128 | 0 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 100 | 0 | 0 | 8064 | Procs | 1 | 1 | 8 | 1 | 1 | 1 |
| | Uncapped | | | Proc Units | 100 | 10 | 800 | 100 | 100 | 100 |
| | | | | Weight | 128 | | | 128 | 128 | 128 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
| 101 | 0 | 0 | 8065 | Procs | 1 | 1 | 8 | 1 | 1 | 0 |
| | Uncapped | | | Proc Units | 100 | 20 | 800 | 100 | 100 | 0 |
| | | | | Weight | 128 | | | 128 | 128 | 0 |
|------|------------|----------|------|------------+-------+-------+-------+-------+-------+-------|
Section 5 - Memory Details
This section details the amount of memory installed in the Power System and how many LMB’s are allocated to each LPAR. InitLmbs are the number of LMBs allocated when the LPAR started. Min and Max LMBs are the LPAR profiles minimum and maximum memory settings and CurLmbs is the number of LBMs currently assigned to the LPAR.
LMB Size.....................256 <== Size of each LBM in Megabytes.
Amount of Licensed Memory....4096 <== Size of Licensed Memory in the Power System.
Licensed Memory in Use.......1813 <== Number of LMBs assigned to LPARs.
LpId HvLpEntry@ Group InitLmbs MinLmbs MaxLmbs CurLmbs TargetLmbs UsableLmbs
0 8000000016D87980 *None 1 1 4096 100 101 2
Phyp-D 1 1
1 8000000076357980 8001 16 8 4000 16 16 1
2 8000000074E77980 8002 48 24 96 48 48 48
3 8000000074F17980 8003 48 24 96 48 48 48
70 80000000A8D57980 8046 0 0 0 0 0 0
71 80000000A93B7980 8047 0 0 0 0 0 0
100 8000000075B47980 8064 32 16 64 32 32 32
101 80000000AA857980 8065 32 16 64 32 32 1
Section 6 - LPAR Core and Memory Affinity
The 2nd to last section of the Resource Dump is probably the most important section as it details your LPARs Core and Memory affinity and the detail contained in this report can be aligned with the ‘lssrad -av’ output from your LPARs to show you where your Virtual Processors are, but most importantly, this section shows you where your Core Entitlement is.
|-----------|-----------------------|---------------|------|---------------|---------------|-------|
| Domain | Procs Units | Memory | | Proc Units | Memory | Ratio |
| SEC | PRI | Total | Free | Free | Total | Free | LP | Tgt | Aloc | Tgt | Aloc | |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
| 0 | | 1200 | 200 | 200 | 2048 | 1813 | | | | | | 5311 |
| | 0 | 600 | 200 | 200 | 1024 | 948 | | | | | | 2777 |
| | | | | | | | 100 | 100 | 100 | 32 | 32 | |
| | | | | | | | 101 | 100 | 100 | 32 | 32 | |
| | 1 | 600 | 0 | 0 | 1024 | 865 | | | | | | 0 |
| | | | | | | | 1 | 100 | 100 | 16 | 16 | |
| | | | | | | | 2 | 400 | 400 | 48 | 48 | |
| | | | | | | | 3 | 100 | 100 | 48 | 48 | |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
| 2 | | 1200 | 1200 | 0 | 2048 | 2048 | | | | | | 689 |
| | 2 | 600 | 600 | 0 | 1024 | 1024 | | | | | | 703 |
| | 3 | 600 | 600 | 0 | 1024 | 1024 | | | | | | 676 |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
The following headings are referenced to the above extract from left to right, as some columns are repeated for both the CEC (Node) and the LPAR columns.
Domains
SECondary domains are Nodes (CECs) as reported in PowerVP and the REF# field in lssrad -av.
PRImary domains are Chips as reported in PowerVP and the SRAD field in lssrad -av.
Please note the both PowerVP and the lssrad commands primary focus is the Virtual Processor locality to memory for an LPAR, these tools are not focusing on where your Entitlement is.
Procs (Entitlement)
Total Procs are the total number of processing units for this Node (CEC) or Chip. Note that each core has a value of 100 Procs, so 1200 procs is equal to 12 cores. This report is from a p8 Power System that has 2 Chips with 6 cores in each Chip, hence 600 for each Chip.
Free Procs are the number of processing units (cores) that have not been allocated to an LPAR. Note that as LPAR entitlements are allocated to a Chip, the value of the free column decreases.
Free Units are the number of processing units (Entitlement) that has been dynamically added to an LPAR. The Hypervisor appears to allocate the Entitlement to anther Chip when the total LPAR Entitlement exceeds the total number of processing units available.
Memory
Total is the total number of LMBs allocated and available for use for this Chip. The size of each LBM is shown further up the report (Section 5) and is shown in Megabytes: LMB Size.....................256. So in our example, each LBM is 256Megabytes.
Free is the number of free LMBs for this Chip. In the example above, Chips 2 and 3 on Node 2 have all the memory free as no LPARs have been allocated memory close to these Chips. Chip 1 on Node 0 has 865 free LBMs and 112 LMBs allocated to LPARs.
Excluded from the report is the Hypervisor Page Table memory for each LPAR, but these values are included in the calculations for Free LMBs. So the Free LMBs, plus all the LPAR allocations will never add up to the Total LMBs value. From these figures, it is possible to calculate the amount of memory used by the Hypervisor Page Tables for all LPARs.
LP
LP is the Logical Partition ID. Details about the LPARid and which LPAR name it maps to can be found in section 1 of the Hypervisor Resource Dump.
Proc Units (LPAR)
Proc Units for the LPARs is the Entitlement each LPAR has to the cores on each Chip. I have never seen the Tgt (Target) and Aloc (Allocated) values to be different. All of the LPARs have their core Entitlement allocated to a single Chip within the Power System. Note that LPARs are not allocated Entitlement to Cores within a Chip, they are only allocated Entitlement to a Chip.
Memory (LPAR)
Memory for the LPARs is the allocation of memory for each LPAR from the local memory DIMMS to that Chip. I have never seen the Tgt (Target) and Aloc (Allocated) values to be different. All of the LPARs have their memory allocated from the local DIMMs to the Chip that their CPU Entitlement is allocated. This is great for affinity and will result is a 100% DPO score for the LPAR.
|-----------|-----------------------|---------------|------|---------------|---------------|-------|
| Domain | Procs Units | Memory | | Proc Units | Memory | Ratio |
| SEC | PRI | Total | Free | Free | Total | Free | LP | Tgt | Aloc | Tgt | Aloc | |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
| 0 | | 1200 | 200 | 200 | 2048 | 1813 | | | | | | 5311 |
| | 0 | 600 | 200 | 200 | 1024 | 948 | | | | | | 2777 |
| | | | | | | | 100 | 100 | 100 | 32 | 32 | |
| | | | | | | | 101 | 100 | 100 | 32 | 32 | |
| | 1 | 600 | 0 | 0 | 1024 | 865 | | | | | | 0 |
| | | | | | | | 1 | 100 | 100 | 16 | 16 | |
| | | | | | | | 2 | 400 | 400 | 48 | 48 | |
| | | | | | | | 3 | 100 | 100 | 48 | 48 | |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
| 2 | | 1200 | 1200 | 0 | 2048 | 2048 | | | | | | 689 |
| | 2 | 600 | 600 | 0 | 1024 | 1024 | | | | | | 703 |
| | 3 | 600 | 600 | 0 | 1024 | 1024 | | | | | | 676 |
|-----|-----|-------|-------|-------|-------|-------|------|-------|-------|-------|-------|-------|
Section 7 - Detailed LPAR Affinity
I have not been able to find a great deal of information on this section of the Hypervisor Resource Dump. The most useful information I can gather from this section is the DPO affinity score (Scr) shown for each LPAR and details about any affinity groups you have confirgurated and which group the LPAR belongs to.
|-----|-----|-|-|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| | |C|P| Domain 0 | Domain 1 | Domain 2 | Domain 3 | Domain 4 | Domain 5 | Domain 6 | Domain 7 |
| | |O|R| Domain 8 | Domain 9 | Domain 10 | Domain 11 | Domain 12 | Domain 13 | Domain 14 | Domain 15 |
| | |N|I| Domain 16 | Domain 17 | Domain 18 | Domain 19 | Domain 20 | Domain 21 | Domain 22 | Domain 23 |
| | |T| | Domain 24 | Domain 25 | Domain 26 | Domain 27 | Domain 28 | Domain 29 | Domain 30 | Domain 31 |
|-----|-----|/|/|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| | O |S|S| | |H|V|P| | |H|V|P| | |H|V|P| | |H|V|P| | |H|V|P| | |H|V|P| | |H|V|P| | |H|V|P|
| | R |P|E| | |P|P|R| | |P|P|R| | |P|P|R| | |P|P|R| | |P|P|R| | |P|P|R| | |P|P|R| | |P|P|R|
| | D |R|C| | |T|T|E| | |T|T|E| | |T|T|E| | |T|T|E| | |T|T|E| | |T|T|E| | |T|T|E| | |T|T|E|
|Lp | R |D| | PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F| PUs |LMBs | | |F|
|-----|-----|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| 1| 4|C|P| | | | | | 100|00010|X|X|X| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Scr=100.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Wgt= 0.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
|-----|-----|---|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| 2| 1031|C|P| | | | | | 400|00030|X|X|X| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Scr=100.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Wgt= 0.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
|-----|-----|---|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| 3| 1033|C|P| | | | | | 100|00030|X|X|X| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Scr=100.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Wgt= 0.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
|-----|-----|---|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| 100| 1036|C|P| 100|00020|X|X|X| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Scr=100.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Wgt= 0.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
|-----|-----|---|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|
| 101| 1085|C|P| 100|00020|X|X|X| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| Scr=100.00| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| AffGrp= 30| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
|-----|-----|---|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|-----|-----|-|-|-|