Posts Tagged ‘vSphere 8.0’

Workaround and Fix – VMware vRealize Log Insight 8.14.1.0-22806512 to 8.16.0-23264422 upgrade failure

Wednesday, August 14th, 2024

These are my memory dump notes from working with a Failed upgrade, and Snapshot revert failed!

I’ve been meaning to upgrade my VMware vRealize Log Insight 8.14.1.0-22806512 appliance in the #homelab for a while, so I was surprised at first when trying to simply upgrade the PAK file it failed with not enough storage in /tmp, so I tried both these updates

  • VMware-vRealize-Log-Insight-8.16.0-23264422.pak
  • VMware-vRealize-Log-Insight-8.16.0-23364779.pak

but failed, so after SSHing into the appliance and checking all the storage, and removing older log files, I noticed that /tmp is defined as a “ram drive”.

So I increased the memory in the Appliance by 2GB, shutdown, changed the memory, and powered on. SSHed back to the appliance

SSH loginisght storage space

SSH log insight storage space

Using the command 

mount -o remount,size=5G /tmp/

Increased the size to 5GB, so at least the upgrade could complete correctly, I also used 

tail -f /storage/var/loginsight/upgrade.log to check the upgrade status, but trying to apply the Scheme upgrade to Cassandra it failed, this is a stock VMware vRealize Log Insight 8.14.1.0-22806512, so not sure why it failed, and there is not much info on the Broadcom site about VMware vRealize Log Insight.

The GUI stated the Upgrade had failed, and although it stated reverting to 8.14.1.0-22806512 it failed.

So I reverted to the snapshot, I had taken before applying the upgrade, but to my surprise, reverting to a snapshot ended up with a completely non-working appliance, the GUI stopped responding. So reaching out for the backups to restore the appliance, quickly found this VM was missing from the jobs!

Duh! Oh shite, got to fix the appliance now, and this is how I fixed it.

1.SSH to appliance

2. service loginsight stop (be prepared to wait a long time!)

3. /usr/lib/loginsight/application/sbin/li-cassandra.sh –startnow –force

it will respond with

Running Operations for Logs stop……..done
Starting Cassandra…..done

WARNING: Be sure to stop Cassandra before attempting to start Operations for Logs!
In worst case, restart the virtual appliance.

4. nodetool-no-pass flush

5. nodetool-no-pass repair –full

6. /usr/lib/loginsight/application/sbin/li-cassandra.sh –stopnow –force

7. service loginsight start

At this point I still did not have a working VMware vRealize Log Insight 8.14.1.0-22806512.

So I then applied the PAK manually.

8. /usr/lib/loginsight/application/sbin/loginsight-pak-upgrade.py /tmp/VMware-vRealize-Log-Insight-8.16.0-23364779.pak (this was already uploaded via WinSCP to /tmp)

and wait…Log Insight Upgrade

There’s alot of changes in the appliance from 8.14 to 8.16 as you can see above!

and now

Log Insight 8.16

Log Insight 8.16

 

 

 

 

 

Log Insight 8.16

Log Insight 8.16

So I hope my notes helps you upgrade your Appliance if you get stuck!

Oh, and I’ve added the VM to a backup job just in case for future, so snapshots don’t always save you!

 

 

Post to Twitter