I have been planning and documenting the upgrade process from AIX 6.1 TL9 SP6 to AIX 7.1 and AIX 7.2. I have seen a strange issue with multibos instances when I attempt to remove the standby instance, the error seen is:
Removing all standby BOS file systems ...
Removing standby BOS file system /bos_inst/opt
0516-1008 rmlv: Logical volume hd10opt must be closed. If the logical
volume contains a filesystem, the umount command will close
the LV device.
multibos: 0565-028 Error removing file systems.
multibos: 0565-002 ATTENTION: cleanup did not complete successfully.
At first I thought that some process had started using /bos_inst/opt but after investigations I found that the cause of the problem is corrupted entries in /etc/filesytems. If you check all the bos_inst entries in /etc/filesystems, you will find they point to the same Logical Volumes as the filesystems without the bos_inst.
E.g. If you grep for opt: in /etc/filesystems, you will see the dev entry is the same for both /opt and /bos_inst/opt. The same is true for the /usr, /var and/.
grep -p opt: /etc/filesystems
/opt:
dev = /dev/hd10opt
vfs = jfs2
log = /dev/hd8
mount = true
check = true
vol = /opt
free = false
quota = no
mb = 25890DCA7B124081
/bos_inst/opt:
dev = /dev/hd10opt
vfs = jfs2
log = /dev/hd8
mount = false
quota = no
mb = 25890DCA7B43FF11
account = false
Here is how to work around the issue, until such time as IBM can fix this.
Run lsvg -l rootvg and confirm that the /bos_inst Logical Volumes are not referencing any filesystems.
lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
bos_hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 64 64 1 open/syncd N/A
hd5 boot 1 1 1 closed/syncd N/A
hd8 jfs2log 1 1 1 open/syncd N/A
hd3 jfs2 48 48 1 open/syncd /tmp
hd1 jfs2 112 112 1 open/syncd /home
hd11admin jfs2 2 2 1 open/syncd /admin
livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump
pridumplv sysdump 10 10 1 open/syncd N/A
bos_hd4 jfs2 16 16 1 closed/syncd N/A
bos_hd2 jfs2 98 98 1 closed/syncd N/A
bos_hd9var jfs2 48 48 1 closed/syncd N/A
bos_hd10opt jfs2 48 48 1 closed/syncd N/A
hd4 jfs2 16 16 1 open/syncd /
hd2 jfs2 98 98 1 open/syncd /usr
hd9var jfs2 48 48 1 open/syncd /var
hd10opt jfs2 48 48 1 open/syncd /opt
Manually edit /etc/filesystems with vi (or your favorite editor) and delete all the bos_inst entries.
You must also remove all the mb = lines from all filesystems. E.g. Remove all the lines like mb = 25890DCA7B124081
Remove all the Logical Volumes that were associated with the bos_inst filesystems.
rmlv -f bos_hd4
rmlv -f bos_hd2
rmlv -f bos_hd9var
rmlv -f bos_hd10opt
rmlv -f bos_hd5
Do not run the chpv -c <diskname> as suggested by the output from rmlv -f bos_hd5, otherwise your system will not boot.
Remove the multibos entry from /etc/inittab.
lsitab -a | grep multi
mbverify:23456789:wait:/usr/sbin/multibos -V 2>&1 | /usr/bin/alog -t boot >/dev/console
rmitab mbverify
Remove the multibos data.
cd /etc/multibos/data
ls -l
total 40
-rw------- 1 root system 17 Dec 14 17:10 acttag
-rw------- 1 root system 27 Dec 13 17:28 curlvlist
-rw------- 1 root system 52 Dec 13 17:28 sbyfslist
-rw------- 1 root system 47 Dec 13 17:28 sbylvlist
-rw------- 1 root system 17 Dec 14 17:10 sbytag
rm /etc/multibos/data/*
Remove the /bos_inst directory if it still exists.
rmdir /bos_inst
Confirm that the multibos instances have been removed.
multibos -R
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...
multibos: 0565-077 Unable to locate standby BOS.
Log file is /etc/multibos/logs/op.alog
Return Status: FAILURE
A multibos setup preview should complete successfully now.
multibos -stp
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...
+-----------------------------------------------------------------------------+
Preview
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
Processing preview information ...
## SNIP ##
Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS
Hi Steve,
Good Day!!!
Could you please help me, with the below multibos, Technical issue
After TL Upgrade with the multibos, while removing the old environment with multibos -R, i am getting the below Error. Could you please help me how to fix this issue.
7100-04-02-1614
And the server configuration is like below now. Please help me.
1.
# multibos -R
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...
+-----------------------------------------------------------------------------+
Remove Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
+-----------------------------------------------------------------------------+
Boot Partition Processing
+-----------------------------------------------------------------------------+
Active boot logical volume is bos_hd5.
Standby boot logical volume is hd5.
+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Unmounting all standby BOS file systems ...
+-----------------------------------------------------------------------------+
File Systems
+-----------------------------------------------------------------------------+
Removing all standby BOS file systems ...
Removing standby BOS file system /bos_inst/opt
0516-1008 rmlv: Logical volume bos_hd10opt must be closed. If the logical volume
contains a filesystem, the umount command will close the LV device.
multibos: 0565-028 Error removing file systems.
multibos: 0565-002 ATTENTION: cleanup did not complete successfully.
Log file is /etc/multibos/logs/op.alog
Return Status: FAILURE
2.
# oslevel -s
7100-04-02-1614
#
#bootlist -m nromal -o
hdisk0 blv=bos_hd5 pathid=0
hdisk0 blv=bos_hd5 pathid=1
hdisko pathid=0
hdisko pathid=1
# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/bos_hd4 0.56 0.23 60% 21472 27% /
/dev/bos_hd2 5.41 1.72 69% 74921 15% /usr
/dev/bos_hd9var 0.75 0.26 65% 7236 10% /var
/dev/hd3 1.00 0.59 42% 635 1% /tmp
/dev/hd1 0.25 0.25 1% 348 1% /home
/dev/hd11admin 0.12 0.12 1% 9 1% /admin
/proc - - - - - /proc
/dev/bos_hd10opt 1.38 0.80 42% 8015 4% /opt
/dev/livedump 0.25 0.25 1% 4 1% /var/adm/ras/livedump
/dev/lv_doonce 0.19 0.01 95% 453 15% /opt/DoOnceAIX
/dev/lv_auditlog 0.75 0.54 29% 2167 2% /var/log/eprise
/dev/perfmgrlv 0.06 0.04 30% 71 1% /var/adm/perfmgr
/dev/usrlocallv 0.12 0.08 40% 219 2% /usr/local
/dev/varlogslv 0.50 0.49 3% 661 1% /var/logs
/dev/varsoelv 0.25 0.24 3% 26 1% /var/soe
/dev/backuplv 2.00 1.99 1% 54 1% /backup
/dev/esdlv 0.50 0.50 1% 5 1% /esd
/dev/uxlv 0.50 0.49 2% 984 1% /var/opt/ux
/dev/seoslv 2.00 1.60 21% 2759 1% /usr/seos
/dev/vartmplv 0.25 0.25 1% 10 1% /var/tmp
/dev/scmlv 0.25 0.21 18% 485 1% /opt/IBM/SCM
/dev/fslv00 0.50 0.49 3% 113 1% /opt/Tivoli
/dev/fslv01 3.00 2.50 17% 3891 1% /opt/IBM/ITM
/dev/fslv02 2.00 1.95 3% 151 1% /opt/IBM/ITM/logs
/dev/fslv03 1.00 0.90 10% 27 1% /opt/IBM/ITM/history
/dev/fslv04 0.25 0.25 1% 23 1% /var/itlm
/dev/temdatalv 3.00 3.00 1% 26 1% /var/opt/BESClient
/dev/temlv 0.12 0.06 49% 16 1% /opt/BESClient
/dev/umilv 12.00 5.13 58% 23 1% /umi
/dev/fslv05 1.00 1.00 1% 5 1% /build_tmp
/dev/gpfs 2201.07 2020.47 9% 5446 1% /gpfs
/dev/gpfslc1 100.05 85.72 15% 4383 2% /MQHA