Install the bos.rte.install fileset from your NIM server:

Preview:

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL1SP2 -a filesets=bos.rte.install -a installp_flags=-pacNYgX

Actual Install:

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL1SP2 -a filesets=bos.rte.install -a installp_flags=-acNYgX

Patch the server to AIX7.2 TL1 SP2, server must already be running AIX7.2.

Preview:

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL1SP2 -a fixes=update_all -a installp_flags=-pacNYgX

Actual Patching:

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL1SP2 -a fixes=update_all -a installp_flags=-acNYgX

Patching a specific set of files.

Generate a listing of filesets that are lower than the current known maintenance level.

KNOWN_OSLEVEL=$( oslevel -sq 2>&1 | tail +3 | head -1 )

oslevel -s -l ${KNOWN_OSLEVEL} 2>&1 | \
grep -e COMMITTED -e APPLIED | \
awk '{ print $1 }' | \
sort -u >> /tmp/filelist

Using NIM, we can install those filesets from above.

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL0SP0-DVD -a filesets=-f/tmp/filelist -a installp_flags=-acNgX

Then update them to the latest TL and SP:

/usr/sbin/nimclient -o cust -a lpp_source=LPP-AIX72TL1SP1 -a fixes=update_all -a installp_flags=-acNYgX

Create an LPP_SOURCE: 

nim -o define -t lpp_source -a server=master -a location=/export/lpp_source/aix72_lpp -a comments="AIX 7.2 lpp source" aix72_lpp

Create a SPOT: 

nim -o define -t spot -a server=master -a location=/export/spot -a comments="AIX 7.2 spot" aix72_spot

Create a MKSYSB definition from existing file: 

nim -o define -t mksysb -a server=master -a location=/export/mksysb/aix72_mksysb -a comments="AIX 7.2 mksysb" aix72_mksysb

These are some usful lslpp commands to find the multiple versions installed, or no longer installed as the case is now.

# lslpp -R ALL -l mqm.java.rte

+-----------------------------------------------------------------------------+
INSTALL ROOT PATH = /usr/mqm71
+-----------------------------------------------------------------------------+

lslpp: 0504-132 Fileset mqm.java.rte not installed.

+-----------------------------------------------------------------------------+
INSTALL ROOT PATH = /usr/mqm9
+-----------------------------------------------------------------------------+

Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/mqm9/usr/lib/objrepos
mqm.java.rte 9.0.0.5 APPLIED IBM MQ Java Client, JMS and
Web Services support

+-----------------------------------------------------------------------------+
INSTALL ROOT PATH = /
+-----------------------------------------------------------------------------+

lslpp: 0504-132 Fileset mqm.java.rte not installed.

While the version of PowerHA is old now, I thought I would still blog this in case anyone has a similar issue.

I found another reference to this issue here:  Powerha-holds-my-disks

EMC Device Details

-------------------------------------------------------------------------
DEVICE         :VEND    :PROD            :REV   :SER NUM          :CAP(kb)
-------------------------------------------------------------------------
/dev/rhdisk12  :EMC     :SYMMETRIX       :5876  :24059ed000 :    11784960

node4:/# lsdev -Cc disk | grep hdisk12

hdisk12  Available 31-T1-01 EMC Symmetrix FCP MPIO VRAID

No volume group on the device.

node4:/# lspv | grep -w -f /tmp/Disk.List.txt

hdisk12         00abc7a7abc2dabd                    None

Current Software Levels.

node4:/# oslevel -s

7100-05-01-1731

node4:/# halevel -s

7.1.3 SP7

Removing the device gives an error.

node4:/# rmdev -l hdisk12

Method error (/etc/methods/ucfgdevice):

        0514-062 Cannot perform the requested function because the

                 specified device is busy.

The device is not in a Fence Group and we appear to have read/write access.

node4:/# /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk12

Disk name:                      hdisk12

Disk UUID:                      f671eedf241cc9cd 0265e4f89c1f3fef

Fence Group UUID:               0000000000000000 0000000000000000 - Not in a Fence Group

Disk device major/minor number: 19, 101

Fence height:                   0 (Read/Write)

Reserve mode:                   0 (No Reserve)

Disk Type:                      0x01 (Local access only)

Disk State:                     17

To fix this, I needed to take the cluster offline and then back online, just on this node.  Note that I used the unmanaged option to avoid an application outage.

node4:/# clmgr offline node node4 when=now manage=unmanage broadcast=false

node4:/# clmgr online node node4 when=now manage=auto broadcast=false

Once the cluster software is up and running, I can delete the device.

node4:/# rmdev -dl hdisk12

hdisk12

hdisk12 deleted

We have already enabled longer passwords in AIX and updated the password algorithm to ssha512.

chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm=ssha512

As all the AIX systems (and other UNIX) have openssl installed, to avoid installing any other tools, we will use openssl.

Generate a 16 character password.

/usr/bin/openssl rand -base64 16

b4all5h6+sLvYh1TPl13yw==

Now putting that in to a script is very easy.  First we generate the password, and remove any $ symbols, as these would be interated by the shell as variables, then display the password to the user and final using chpasswd we can apply the new random password to username.

NEWPASS=$( /usr/bin/openssl rand -base64 16 | tr -d "$" )

print -- ${NEWPASS}

print -- username:${NEWPASS} | /usr/bin/chpasswd

This may or may not be a security concern to you, but I would prefer each of my VIO Servers to have unique host keys.  I believe it is best practice for security sake to have every server have a unique SSH host key.

# /usr/ios/cli/ioscli ioslevel
3.1.0.00

SSH Host Keys

By default, the install just uses the SSH host keys already in the mksysb image.

# cd etc/ssh

# ls -l ssh_host*

-rw------- 1 root system  668 19 Oct 06:49 ssh_host_dsa_key
-rw-r--r-- 1 root system  590 19 Oct 06:49 ssh_host_dsa_key.pub
-rw------- 1 root system  227 19 Oct 06:49 ssh_host_ecdsa_key
-rw-r--r-- 1 root system  162 19 Oct 06:49 ssh_host_ecdsa_key.pub
-rw------- 1 root system  387 19 Oct 06:49 ssh_host_ed25519_key
-rw-r--r-- 1 root system   82 19 Oct 06:49 ssh_host_ed25519_key.pub
-rw------- 1 root system 1679 19 Oct 06:49 ssh_host_rsa_key
-rw-r--r-- 1 root system  382 19 Oct 06:49 ssh_host_rsa_key.pub

Let's take a look at one of those keys.

Does the public key look exactly the same as the one on your VIO Server?

# cat ssh_host_rsa_key.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEpdzANZeF9ul8KOENb6FO/Iu4CznbO49jaBhS8vFRtwwT/
0Y+VoSCIPc/BVPGUYzdu1e27Y4kTt1cFhQgxOsXaARgQMplYJ5F8UciiJSTujVQzQbJhYMk6VISmdEQ/HINIG
SeI+hyIQZ9fnAlo0HrjVzIK0EPVKVV5Wsu63CtA3PhE4ONr10BZBkUzwV0+7KFOecQ0KjDchHJeM1GeqVrqWO
hqKTy36Zg8bXsCgwfPPKh6JZGuimPrQ9CTvhBGXyNEw2USO1Z4lXtbc+z7aFdGtmamkvWE2cFD190uaQoqK9o
Yj7VuhCD/5J0WnkOdjpyQ73LBhTxsugRFb2wcejP

To Fix This.

# rm /etc/ssh/ssh_host*

# ssh-keygen -P '' -t rsa -f /etc/ssh/ssh_host_rsa_key

# /usr/ios/cli/ioscli ioslevel
3.1.0.00

Paging Devices

By default, the install creates a hd6 at 512MB and a paging00 device at 1024MB.  The AIX best practice guides advise against this.

# lsps -a
Page Space Physical Volume Volume Group  Size    %Used  Active  Auto  Type  Chksum
paging00   hdisk0   rootvg               1024MB      1    yes    yes   lv      0
hd6        hdisk1   rootvg                512MB      0    yes    yes   lv      0

Filesystems maybe too small.

These filesystems maybe on the small size.  I like to have some space for the logging in /var, so I make this 2GB and I like / to be 1GB, just in case!!

# df -g
Filesystem   GB blocks  Free  %Used  Iused  %Iused  Mounted on
/dev/hd4        0.50    0.45    10%   4139     4%   /
/dev/hd9var     1.00    0.77    23%   3630     2%   /var
 

AIX and VIOS still use crypt for password hashing.

This only enforces the first 8 characters of the password.  I highly recommend you change this to sha512.

Before, you will see that pwd_algorithm is blank.

# lssec -f /etc/security/login.cfg -s usw -a pwd_algorithm
usw pwd_algorithm=

Change the password encryption to sha512.

# chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm=ssha512

After the change, confirm it is now sha512.

# lssec -f /etc/security/login.cfg -s usw -a pwd_algorithm
usw pwd_algorithm=ssha512

From the man page for vmstat it states:

If the current physical processor consumption of the uncapped partitions exceeds the entitled capacity, the percentage becomes relative to the number of physical processor consumed (pc).

What the man page fails to state is that if the current physical processor consumption of the uncapped partition is less than the entitled capacity, the percentage is in relation to the entitled capacity of the LPAR.

The idle system - no workload except vmstat running.

Note the LPAR has 4 Virtual Processors and starts with an Entitlement of 1 core.

r b p w   avm     fre  fi fo pi po fr sr in  sy  cs us sy id wa  pc  ec  hr mi se
0 0 0 0 443929 1585346 0  0  0  0  0  0   1 372 173  0  0 99  0 0.01 0.8 19:18:09
0 0 0 0 443930 1585345 0  0  0  0  0  0   0  50 167  0  0 99  0 0.00 0.4 19:18:10
0 0 0 0 443930 1585345 0  0  0  0  0  0   2 100 206  0  0 99  0 0.01 0.5 19:18:11
0 0 0 0 443930 1585345 0  0  0  0  0  0   5  50 169  0  0 99  0 0.00 0.4 19:18:12


Now starting 1 spinning CPU thread.
So a single thread running gives the 59 user and 41 idle percentages as expected for p8.

1 0 0 0 443978 1585296 0  0  0  0  0  0   1  47 167 59  0 41  0 1.00 100.3 19:18:50
1 0 0 0 443978 1585296 0  0  0  0  0  0   5  99 207 59  0 41  0 1.00 100.4 19:18:51
1 0 0 0 443978 1585296 0  0  0  0  0  0   1  56 174 59  0 41  0 1.01 100.5 19:18:52
1 0 0 0 443978 1585296 0  0  0  0  0  0   1  47 169 59  0 41  0 1.00 100.2 19:18:53


Now I add more Entitlement using DLPAR.
Upping the Entitlement to 2 changes the Entitled percentage from 100% to 50%, but does not change the PC column, we are still running a single spinning thread on 1 CPU.  Yet, user drops to 30%, half of the 60% in the above test.  We just added 100% more Entitlement, and idle increases to 70%, that’s 50% idle for the extra entitlement, plus 20% of the Entitlement that is being used (20% of 2 Entitlement = 40% of 1 Entitlement).
So if the LPAR is below entitlement, these values are percentages of the Entitlement, not the PC column.

1 0 0 0 443988 1585259 0  0  0  0  0  0   0  52 195 30  0 70  0 1.00 50.2 19:19:37
1 0 0 0 443988 1585259 0  0  0  0  0  0   1  82 186 30  0 70  0 1.00 50.2 19:19:38
1 0 0 0 443988 1585259 0  0  0  0  0  0   0  60 183 30  0 70  0 1.00 50.2 19:19:39
1 0 0 0 443988 1585259 0  0  0  0  0  0   0  55 184 30  0 70  0 1.00 50.2 19:19:40

So now I add another spinning thread and we go back to the expected 59% user and 41% idle.  Why?  Well both cores are running a single thread on the primary SMT, so we get the expected 59% user and 41% idle as per p8.  As the LPAR is now running above entitlement, the user, system, wait and idle percentages are now in relation to the PC column.

2 0 0 0 444053 1585188 0  0  0  0  0  0   2  70 170 59  0 41  0 2.00 100.2 19:20:14
2 0 0 0 444053 1585188 0  0  0  0  0  0   0  46 165 59  0 41  0 2.00 100.1 19:20:15
2 0 0 0 444053 1585188 0  0  0  0  0  0   0 128 178 59  0 41  0 2.00 100.2 19:20:16
2 0 0 0 444053 1585188 0  0  0  0  0  0   2  47 169 59  0 41  0 2.00 100.1 19:20:17

Adding another thread does not change the percentage values for user and idle, as the new core is running only 1 SMT thread.

3 0 0 0 436105 1593130 0  0  0  0  0  0   5  46 165 59  0 41  0 3.00 150.0 19:20:39
3 0 0 0 436105 1593130 0  0  0  0  0  0   9  56 167 59  0 41  0 3.00 150.0 19:20:40
3 0 0 0 436105 1593130 0  0  0  0  0  0   1  58 201 59  0 41  0 3.02 151.0 19:20:41
3 0 0 0 436105 1593130 0  0  0  0  0  0   2  47 167 60  0 40  0 2.98 149.1 19:20:42