These are my memory dump notes from working with a Failed upgrade, and Snapshot revert failed!
I’ve been meaning to upgrade my VMware vRealize Log Insight 8.14.1.0-22806512 appliance in the #homelab for a while, so I was surprised at first when trying to simply upgrade the PAK file it failed with not enough storage in /tmp, so I tried both these updates
- VMware-vRealize-Log-Insight-8.16.0-23264422.pak
- VMware-vRealize-Log-Insight-8.16.0-23364779.pak
but failed, so after SSHing into the appliance and checking all the storage, and removing older log files, I noticed that /tmp is defined as a “ram drive”.
So I increased the memory in the Appliance by 2GB, shutdown, changed the memory, and powered on. SSHed back to the appliance
Using the command
mount -o remount,size=5G /tmp/
Increased the size to 5GB, so at least the upgrade could complete correctly, I also used
tail -f /storage/var/loginsight/upgrade.log to check the upgrade status, but trying to apply the Scheme upgrade to Cassandra it failed, this is a stock VMware vRealize Log Insight 8.14.1.0-22806512, so not sure why it failed, and there is not much info on the Broadcom site about VMware vRealize Log Insight.
The GUI stated the Upgrade had failed, and although it stated reverting to 8.14.1.0-22806512 it failed.
So I reverted to the snapshot, I had taken before applying the upgrade, but to my surprise, reverting to a snapshot ended up with a completely non-working appliance, the GUI stopped responding. So reaching out for the backups to restore the appliance, quickly found this VM was missing from the jobs!
Duh! Oh shite, got to fix the appliance now, and this is how I fixed it.
1.SSH to appliance
2. service loginsight stop (be prepared to wait a long time!)
3. /usr/lib/loginsight/application/sbin/li-cassandra.sh –startnow –force
it will respond with
Running Operations for Logs stop……..done
Starting Cassandra…..done
WARNING: Be sure to stop Cassandra before attempting to start Operations for Logs!
In worst case, restart the virtual appliance.
4. nodetool-no-pass flush
5. nodetool-no-pass repair –full
6. /usr/lib/loginsight/application/sbin/li-cassandra.sh –stopnow –force
7. service loginsight start
At this point I still did not have a working VMware vRealize Log Insight 8.14.1.0-22806512.
So I then applied the PAK manually.
8. /usr/lib/loginsight/application/sbin/loginsight-pak-upgrade.py /tmp/VMware-vRealize-Log-Insight-8.16.0-23364779.pak (this was already uploaded via WinSCP to /tmp)
There’s alot of changes in the appliance from 8.14 to 8.16 as you can see above!
and now
So I hope my notes helps you upgrade your Appliance if you get stuck!
Oh, and I’ve added the VM to a backup job just in case for future, so snapshots don’t always save you!