Archive for the ‘ExpertsExchange’ Category

HOW TO: Check if you have compromised your VMware ESXi 8.0 Hosts if you have added them to Microsoft Active Directory

Monday, August 26th, 2024

This video was created in response to Experts Exchange members asking the question “have I compromised my ESXi host be adding to AD?”

In this video presentation which is part of the Hancock’s VMware Half Hour I will show you HOW TO: Check if you have compromised your VMware ESXi 8.0 Hosts if you have added them to Microsoft Active Directory.

In this video demonstration the ESXi servers are ESXi 8.0.3, which have the “fix” detailed below

Secure Default Settings for ESXi Active Directory integration

To demonstrate the differences between a compromised and non-compromised server, I have deliberately changed the default settings on esxi002.cyrus-consultants.co.uk, so the server can be compromised.
HOW NOT TO: Compromise your VMware vSphere Hypervisor ESXi 5.1, 5.5, 6.0, 6.5, 6.7, 7.0, 8.0 by adding to Microsoft Active Directory

On the 29th July 2024, Microsoft  discovered a vulnerability in ESXi hypervisors being exploited by several ransomware operators to obtain full administrative permissions on domain-joined ESXi hypervisors.

this publication is here – https://www.microsoft.com/en-us/security/blog/2024/07/29/ransomware-operators-exploit-esxi-hypervisor-vulnerability-for-mass-encryption/

VMware vExperts – Christian Mohn wrote about it here – VMware vSphere CVE-2024-37085 – A Nothing Burger

and Bob Plankers goes into more detail here – Thoughts on CVE-2024-37085 & VMSA-2024-0013

Please have a read of these publications.

Broadcom have issued updates and fixes to vSphere 7.0 and 8.0, and VCF 4.x and 5.x only. There is no security update for 6.7.

https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/24505

HOW TO: Update VMware vSphere vCenter Server 8.0.3 to 8.0.3a using the Reduced Downtime Upgrade (RDU) function

Monday, August 26th, 2024

VMware vCenter Server 8.0.3a  Build 24091160 was released on the 17th July 2024.

In this video presentation which is part of the Hancock’s VMware Half Hour I will show you HOW TO: Update VMware vSphere vCenter Server 8.0.3 to 8.0.3a using the Reduced Downtime Upgrade (RDU) function.

This procedure can be used to update any VMware vCenter Server 8.0 update in the future. VMware have released a new feature in later versions of VMware vSphere vCenter 8.0 called the Reduced Downtime Upgrade (RDU), which reduces downtime from over 60, minutes to 10 minutes. We would traditionally use the VAMI method here

HOW TO: Update VMware vSphere vCenter Server 8.0.2 (8.0u2d) 8.0.2 to 8.0.3 using the (VAMI) Appliance Management Interface

but RDU only takes 15 minutes, but it does require the original vCenter Server 8.0.3 iso media!

At the time of this recording, VMware vCenter Server 8.0 Update 3a was the latest version available from VMware.

VMware vCenter Server 8.0 Update 3a Release Notes

HOW TO: Update VMware vSphere vCenter Server 8.0.2 (8.0u2d) 8.0.2 to 8.0.3 using the Reduced Downtime Upgrade (RDU) function

Videos mentioned in this video, this method can be used to backup the vCenter Server database.

HOW TO: Use the vCenter Server 7.0.3 vCenter Server Appliance Management Interface (VAMI) to backup the database and configuration of your vCenter Server

HOW TO: Restore a vCenter Server backup to restore a production vCenter Server 8.0 appliance

HOW TO: Update VMware ESXi 8.0 GA to ESXi 8.0U2 direct from VMware remotely using the ESXCLI tool installed on Windows 10

HOW TO: Remediate a vSphere Cluster VMware ESXi 8.0U2 to ESXi 8.0U3 including adding the HPE OEM Addon for ESXi 8.0.3 – A12 using VMware vSphere Lifecycle Manager (vLCM) from a single image

HOW TO: Migrate physical, virtual and cloud based workloads with real-time replication to VMware vSphere (ESXi) or Microsoft Hyper-V using OpenText Migrate

Friday, August 23rd, 2024

In this video presentation which is part of the [url=”https://github.com/CodheadClub/AwesomeResources/blob/master/Virtualisation.md#andrewhancock”]Hancock’s VMware Half Hour[/url] I will show you HOW TO: Migrate physical, virtual and cloud based workloads with real-time replication to VMware vSphere (ESXi) or Microsoft Hyper-V using OpenText Migrate.

Ten years ago I wrote the following article about the product Double-Take MOVE by Vision Solutions. It’s been acquired in the last ten years, by Carbonite, and now is in the hand of OpenText, and is name going forward will be [url=”https://www.opentext.com/en-gb/products/migrate”]OpenText Migrate.[/url]

[url=”https://www.experts-exchange.com/articles/17312/HOW-TO-Migrate-physical-virtual-and-cloud-based-workloads-with-real-time-replication-to-VMware-vSphere-ESXi-using-Double-Take-MOVE.html”]HOW TO: Migrate physical, virtual and cloud based workloads with real-time replication to VMware vSphere (ESXi) using Double-Take MOVE[/url]

In these uncertain times, when you may be required to migrate workloads between platforms, such as VMware vSphere, Microsoft Hyper-V etc I can highly recommend [url=”https://www.opentext.com/en-gb/products/migrate”]OpenText Migrate.[/url] as one of the best products on the market for your Migration Project.

In this video I will demonstrate the migration of an Linux Ubuntu 22.04 LTS Server from Microsoft Hyper-V to VMware vSphere, and concurrently migrate a Microsoft Windows 2022 Server from VMware vSphere to Microsoft Hyper-V, with very little downtime.

Time for a facelift and new branding – Hancock’s VMware Half Hour

Thursday, August 22nd, 2024
Hancock's VMware Half Hour

Hancock’s VMware Half Hour

It was time for a change! So from today, I’ll be using this new artwork, across the social channels, and for more VMware content to come. It encompasses three things in my life, which I hold true, which is VMware, Experts Exchange and #beekeeping. My Thanks to Devolutions and their graphic designer Stacy Bensa for all the hard work, in creating the design for me. Again Thank You.

HOW NOT TO: Compromise your VMware vSphere Hypervisor ESXi 5.1, 5.5, 6.0, 6.5, 6.7, 7.0, 8.0 by adding to Microsoft Active Directory

Thursday, August 22nd, 2024

In this video presentation which is part of the Hancock’s VMware Half Hour I will show you HOW NOT TO: Compromise your VMware vSphere Hypervisor ESXi 5.1, 5.5, 6.0, 6.5, 6.7, 7.0, 8.0 by adding to Microsoft Active Directory. I will demonstrate the exposure, and discuss how to avoid it.

In this video demonstration the ESXi servers are ESXi ARM 7.0, but the same functionality has been built into ESXi since 5.1.

On the 29th July 2024, Microsoft  discovered a vulnerability in ESXi hypervisors being exploited by several ransomware operators to obtain full administrative permissions on domain-joined ESXi hypervisors.

this publication is here – https://www.microsoft.com/en-us/security/blog/2024/07/29/ransomware-operators-exploit-esxi-hypervisor-vulnerability-for-mass-encryption/

VMware vExperts – Christian Mohn wrote about it here – VMware vSphere CVE-2024-37085 – A Nothing Burger

and Bob Plankers goes into more detail here – Thoughts on CVE-2024-37085 & VMSA-2024-0013

Please have a read of these publications.

Broadcom have issued updates and fixes to vSphere 7.0 and 8.0, and VCF 4.x and 5.x only. There is no security update for 6.7.

https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/24505

He’s like a VMware Yoda! :)

Wednesday, August 21st, 2024

A recent testimonial on Experts Exchange

VMware Yoda

VMware Yoda

A VMware Yoda !

A VMware Yoda !

HOW TO: Fix Synchronous Exception at 0x00000000XXXXXXX on VMware vSphere Hypervisor 7.0 (ESXi 7.0 ARM) on a Raspberry Pi 4

Tuesday, August 20th, 2024

In this video presentation which is part of the Hancock’s VMware Half Hour I will show you HOW TO: Fix Synchronous Exception at 0x00000000XXXXXXX on VMware vSphere Hypervisor 7.0 (ESXi 7.0 ARM) on a Raspberry Pi 4.

It has been well documented that the Raspberry Pi 4 UEFI Firmware Image can cause this fault which renders the UEFI boot image corrupt. See here https://github.com/pftf/RPi4/issues/97

The UEFI firmware imaged used in the lab in this video is v1.37, it is debated as too whether this has been fixed in later releases v1.37, some suggest rolling back to v1.33 !

For the sake of continuity I’ve included previous EE Videos and Articles I’ve created here

Part 51. HOW TO: Update the VMware vSphere Hypervisor 7.0 ARM Edition (ESXi 7.0 ARM edition) from v1.12 to v1.15 on a Raspberry Pi 4

Part 20: HOW TO: Install and Configure VMware vSphere Hypervisor 7.0 (ESXi 7.0 ARM) on a Raspberry Pi 4

Part 23: HOW TO: BOOT VMware vSphere Hypervisor 7.0 (ESXi 7.0 ARM) from an iSCSI LUN for the Raspberry Pi 4

Workaround and Fix – VMware vRealize Log Insight 8.14.1.0-22806512 to 8.16.0-23264422 upgrade failure

Wednesday, August 14th, 2024

These are my memory dump notes from working with a Failed upgrade, and Snapshot revert failed!

I’ve been meaning to upgrade my VMware vRealize Log Insight 8.14.1.0-22806512 appliance in the #homelab for a while, so I was surprised at first when trying to simply upgrade the PAK file it failed with not enough storage in /tmp, so I tried both these updates

  • VMware-vRealize-Log-Insight-8.16.0-23264422.pak
  • VMware-vRealize-Log-Insight-8.16.0-23364779.pak

but failed, so after SSHing into the appliance and checking all the storage, and removing older log files, I noticed that /tmp is defined as a “ram drive”.

So I increased the memory in the Appliance by 2GB, shutdown, changed the memory, and powered on. SSHed back to the appliance

SSH loginisght storage space

SSH log insight storage space

Using the command 

mount -o remount,size=5G /tmp/

Increased the size to 5GB, so at least the upgrade could complete correctly, I also used 

tail -f /storage/var/loginsight/upgrade.log to check the upgrade status, but trying to apply the Scheme upgrade to Cassandra it failed, this is a stock VMware vRealize Log Insight 8.14.1.0-22806512, so not sure why it failed, and there is not much info on the Broadcom site about VMware vRealize Log Insight.

The GUI stated the Upgrade had failed, and although it stated reverting to 8.14.1.0-22806512 it failed.

So I reverted to the snapshot, I had taken before applying the upgrade, but to my surprise, reverting to a snapshot ended up with a completely non-working appliance, the GUI stopped responding. So reaching out for the backups to restore the appliance, quickly found this VM was missing from the jobs!

Duh! Oh shite, got to fix the appliance now, and this is how I fixed it.

1.SSH to appliance

2. service loginsight stop (be prepared to wait a long time!)

3. /usr/lib/loginsight/application/sbin/li-cassandra.sh –startnow –force

it will respond with

Running Operations for Logs stop……..done
Starting Cassandra…..done

WARNING: Be sure to stop Cassandra before attempting to start Operations for Logs!
In worst case, restart the virtual appliance.

4. nodetool-no-pass flush

5. nodetool-no-pass repair –full

6. /usr/lib/loginsight/application/sbin/li-cassandra.sh –stopnow –force

7. service loginsight start

At this point I still did not have a working VMware vRealize Log Insight 8.14.1.0-22806512.

So I then applied the PAK manually.

8. /usr/lib/loginsight/application/sbin/loginsight-pak-upgrade.py /tmp/VMware-vRealize-Log-Insight-8.16.0-23364779.pak (this was already uploaded via WinSCP to /tmp)

and wait…Log Insight Upgrade

There’s alot of changes in the appliance from 8.14 to 8.16 as you can see above!

and now

Log Insight 8.16

Log Insight 8.16

 

 

 

 

 

Log Insight 8.16

Log Insight 8.16

So I hope my notes helps you upgrade your Appliance if you get stuck!

Oh, and I’ve added the VM to a backup job just in case for future, so snapshots don’t always save you!

 

 

A list of all my @ExpertsExchange articles and videos can be found at The CodHeadClub

Monday, August 21st, 2023

A list of all my Experts Exchange articles and videos can be found here – at the  – CodHeadClub – To copy and paste! 

http://tinyurl.com/AwesomeResourcesURL

This is an Awesome List of Computer Science, Technology, Programming and Educational resources for the benefit of all who care to use it.

The list was originally created by Closebracket.

I’ve now written over 140 articles and created 40 hours of tutorial VMware vSphere videos on vSphere 7.0 and 8.0. and today published Part 50 – VMware vSphere videos on vSphere 7.0.

HOW TO: Perform storage performance tests on VMware vSphere vSAN, using the VMware Hyper-converged Infrastructure Benchmark fling (HCIBench)

Monday, August 14th, 2023

In this video presentation which is part of the Hancock’s VMware Half Hour HOW TO Video Series I will show you HOW TO:  Perform storage performance tests on VMware vSphere vSAN, using the VMware Hyper-converged Infrastructure Benchmark fling (HCIBench).

HCIBench is a storage performance testing automation tool that simplifies and accelerates customer Proof of Concept (POC) performance testing in a consistent and controlled way. VMware vSAN Community Forum provides support for HCIBench.

HCIBench

The storage devices we are using in this video are the Intel® Optane™ SSD DC P4800X Series 375GB, 2.5in PCIe x4, 3D XPoint™, but this procedure can be use to add any compatible storage devices in ESXi to a vSAN datastore.

This video follows on from the follow video in this series

Part 36: HOW TO: Select an inexpensive HCL Certified 10GBe network interfaces for vSphere ESXi 7.0 and vSphere ESXi 8.0 for VMware vSphere vSAN

Part 37: HOW TO: Change the LBA sector size of storage media to make it compatible with VMware vSphere Hypervisor ESXi 7.0 and ESXi 8.0.

Part 39: HOW TO: Create a VMware vSphere Distributed Switch (VDS) for use with VMware vSphere vSAN for the VMware vSphere vSAN Cluster.

If you are creating a design for VMware vSphere vSAN for a Production environment, please ensure you read the  VMware Cloud Foundation Design Guide 01 JUN 2023 – this should be regarded as The Bible!

References

HOW TO: FIX the Warning System logs on host are stored on non-persistent storage, Move system logs to NFS shared storage.

WHAT’S HAPPENING WITH INTEL OPTANE? – Mr vSAN – Simon Todd

Matt Mancini blog

VMware vSAN 8.0 U1 Express Storage Architecture Deep Dive

VMware vSAN 7.0 U3 Deep Dive Paperback – 5 May 2022

The results generated from this video are available here in these PDFs for download

FIO Benchmarks

4K/70%Read/100%Random

4K/100%Read100%Random

8K/50%Read/100%Random

256K/100%Write/100%Sequential

VDBENCH Benchmarks

4K/70%Read/100%Random
4K/100%Read100%Random
8K/50%Read/100%Random
256K/100%Write/100%Sequential