This video builds on the last videos in this series of Hancock’s VMware Half Hour, where we continue to build our VMware vSphere 7.0 Lab, and in this video, we start to explore the Warning messages appearing on several of our hosts in the cluster.
If you see the Warning message This host is potentially vulnerable to issues described in CVE-2018-3646. please refer to http://kb.vmware.com/s/article/55636 for details and VMware recommendations KB 55636. then this secuerity fix is required for your VMware vSphere Hypervsior (ESXi 7.0) host.
In this video, I will show you HOW TO: FIX VMware vSphere Hypervisor (ESXi) Host vulnerability L1 Terminal Fault’ (L1TF) Speculative-Execution in Intel processors: CVE-2018-3646, CVE-2018-3620, and CVE-2018-3615.
Modern Intel CPU do not suffer this issue, but earlier processors do have these security issues, in March 2018, security issues were detected in Intel processors called Meltdown, Spectre and L1 Terminal Fault.
The previous videos are listed here for your convenience
This video builds on the last videos in this series of Hancock’s VMware Half Hour, where we continue to build our VMware vSphere 7.0 Lab, and in this video, we start to explore the Warning messages appearing on several of our hosts in the cluster.
In this video, I will show you HOW TO: FIX the No coredump target has been configured. Host core dumps cannot be saved. There have been some changes in ESXi 7.0, and the core dumps can now only be written to SSD, HDD, NVMe, not persistent storage such as USB flash drives or SD cards. I discuss some of the issues with ESXi 7.0 and the support of USB flash drives and SD cards. There has been some confusion over the last 18 months about the support of ESXi 7.0 on USB flash drives and SD cards, VMware, Dell and HPE state there were not supported, because 7.0.2 or later caused issues with corruption of the ESXi installation if installed on these devices, and later VMware stated this was deprecated,. and then more recently did an about turn and not it’s supported.
I think the “writing is on the wall” and if you are in a production environment using USB flash drives and SD cards, now is the time to change to SSD, HDD, NVMe or SATADOM for ESXi installations.
The previous videos are listed here for your convenience
This video builds on the last videos in this series of Hancock’s VMware Half Hour, where we continue to build our VMware vSphere 7.0 Lab, and in this video, we start to explore the Warning messages appearing on several of our hosts in the cluster.
In this video, I will show you HOW TO: FIX the Warning System logs on the host are stored on non-persistent storage by moving the system logs to NFS shared storage.
The previous videos are listed here for your convenience
In this video, I will show you “Andy’s Towers of Hanoi solution to Migrating VMware vCenter Server 7.0” – HOW TO: Migrate VMware vCenter Server 7.0 in an enabled EVC VMware vSphere 7.0 Cluster of ESXi 7.0 hosts.
The reason I call this the Andy’s Towers of Hanoi solution is that the Towers of Hanoi is a puzzle game, and a logical approach is required in a certain order to solve it. This is very similar to the puzzle or headache, when you have older and new CPU generation VMware vSphere Hypervisors Hosts (ESXi hosts), and vCenter Server is currently hosted on a new CPU host (with newer CPU), and you want to combine all these hosts in a Cluster using an EVC baseline.
Like the individual blocks of the Towers of Hanoi puzzle, represents a procedure or function in the process to migrate the vCenter Server.
In this tutorial and example in this Lab, this is complicated in the virtual machine vCenter Server 7.0 are also hosted on shared storage which is also hosted on the Dell PowerEdge R730, which we want to add to a Cluster of Dell PowerEdge R720. This example represents adding a new CPU generation host “Haswell” to older CPU generation hosts “Sandy Bridge” in the same Cluster.
In Summary
1. Storage vMotion all virtual machines from shared storage to local storage on the Dell PowerEdge R730.
2. Power off the virtual machine Synology NAS
3. Enable EVC “Sandy Bridge” Per VM EVC for Synology NAS.
4. Power on virtual machine Synology NAS.
5. Storage vMotion all virtual machines from local storage on the Dell PowerEdge R730 to Shared storage on the Synology NAS.
6. Power off DNS virtual machine.
7. Cold Migrate to Cluster. Power on DNS virtual machine.
8. Power off vCenter Server 7.0 virtual machine.
9. Unregister from vCenter Server 7.0 host (Dell PowerEdge R730).
10. Register vCenter Server 7.0 on new host (Dell PowerEdge R720)/
11. Power on vCenter Server 7.0 on new host (Dell PowerEdge R720).
12. Login to vCenter Server 7.0.
13. Drag and Drop Dell Power Edge R730 into Cluster.
14. Done.
In this video, I will show you how to HOW TO: Add a Synology NAS providing NFS Storge to VMware vSphere Hypervisor ESXi 7.0. I will show you the process of creating a storage pool followed by a btrfs volume, enabling NFS and Exporting the NFS volume and connecting to VMware vSphere Hypervisor ESXi 7.0, by creating a dedicated storage network using a VMkernel portgroup for NFS traffic.
I will show you how to troubleshoot and check using simple bash tools ping and vmkping, that NFS traffic can reach the NAS.
The syntax I’m using in this video to test network communications between VMkernel portgroups and NAS (NFS), and check for jumbo frames.
ping <IP address>
vmkping -I vmkX <IP address> -s 9000
where X is a number of your VMkernel portgroup
In this video, I make reference to the previous videos written articles
I’m excited to announce that my entire Hancock’s VMware Half Hour HOW TO: VMware vSphere 7.0 series is now fully uploaded on YouTube! After much effort, the back catalog is finally available, offering nearly 22 hours of in-depth VMware vSphere 7.0 content.
With a total playlist duration of 21 hours, 58 minutes, and 21 seconds, this series is designed to help you dive deep into VMware vSphere 7.0, from foundational concepts to advanced techniques. Whether you’re just getting started or looking to refine your existing skills, this series has something for everyone.
What to Expect:
Concise Sessions: Each video is around 30 minutes long, perfect for those who want quick, focused learning sessions.
Comprehensive Coverage: From installation and configuration to troubleshooting and advanced tips, the series covers the full spectrum of VMware vSphere 7.0 functionality.
Step-by-Step Demos: Many episodes include live demonstrations, making it easy to follow along and implement what you learn.
Why I Created This Series:
VMware vSphere is a robust platform, but mastering it can be a challenge. That’s why I set out to create bite-sized, easily digestible content that fits into a busy schedule. Each episode builds upon the last, so by the end, you’ll have a thorough understanding of VMware vSphere 7.0.
How to Access the Playlist:
The full playlist is available on my YouTube channel. Whether you want to binge-watch or pick a topic that interests you, it’s all at your fingertips. Feel free to share your thoughts, ask questions, or request new topics in the comments section of each video.
Head over to YouTube, grab a coffee, and start learning!
Stay tuned for more exciting content, and thank you to everyone who has supported the channel along the way!
In this video, I will show you how to HOW TO: Add a Synology NAS providing NFS Storge to VMware vSphere Hypervisor ESXi 7.0. I will show you the process of creating a storage pool followed by a btrfs volume, enabling NFS and Exporting the NFS volume and connecting to VMware vSphere Hypervisor ESXi 7.0, by creating a dedicated storage network using a VMkernel portgroup for NFS traffic.
I will show you how to troubleshoot and check using simple bash tools ping and vmkping, that NFS traffic can reach the NAS.
The syntax I’m using in this video to test network communications between VMkernel portgroups and NAS (NFS), and check for jumbo frames.
ping <IP address>
vmkping -I vmkX <IP address> -s 9000
where X is a number of your VMkernel portgroup
In this video, I make reference to the previous videos written articles
In this video, I will show you how to HOW TO: Cross vCenter Server vMotion (export) between standalone vCenter Servers 7.0 not linked to the current SSO domain. I will show you how to troubleshoot and check using simple bash tools ping and vmkping, that it will be possible to perform a Storage vMotion. I will touch briefly on Enhanced vMotion Compatibility (EVC), and migrate and demonstrate live a Storage vMotion between different generation processors, and create an EVC baseline per VM, and end with a Migration (cold).
The syntax I’m using in this video to test network communications between VMkernel portgroups
In today’s dynamic IT landscape, staying up-to-date with the latest tools and upgrades is crucial for maintaining a secure and efficient system. I recently stumbled upon an essential tool for VMware vSphere administrators, and I’m eager to share my experience and insights. Whether you’re a seasoned VMware administrator or just starting out, this guide will walk you through the vital steps and considerations for using VDT and upgrading your vCenter Server.
Unveiling VDT: The vSphere Diagnostic Tool
Introduction to VDT and Its History
Let’s dive into one of the essential tools in the VMware vSphere toolkit: the vSphere Diagnostic Tool (VDT). Formerly known under the same name, it has been a trusted companion for VMware administrators since its introduction in VMware vSphere 6.5. We use it every week.
We first encountered VDT around the time when VMware was gaining immense popularity. Back then, it was just another script in the VMware vSphere arsenal. Fast forward to today, VDT has evolved into a powerful diagnostic tool, and interestingly, the script file is still named VDT.py.
Now, given the updates from Broadcom, everything is transitioning to VCF (VMware Cloud Foundation).
Purpose and Functionality of VDT
Why exactly is VDT so significant? Simply put, it ensures the health and readiness of your VMware vCenter Server environments before you perform an upgrade. VDT checks essential elements—passwords, certificates, storage, etc.—ensuring that you don’t step into avoidable issues.
Imagine you’re about to make a crucial upgrade, and you realize post-upgrade that certificates were expired or storage was misconfigured. Nightmare, right?
VDT is the guardian here, checking things thoroughly so you’re poised for a smooth upgrade process. Let’s look ahead to usage but, first, a quick rundown:
Parameter
Details
Tool Name
VDT (vSphere Diagnostic Tool)
Script File
VDT.py
Introduced in
VMware vSphere 6.5
How to Download and Install VDT
Getting started with VDT is a breeze. Here’s a step-by-step guide:
Click on the Download button to get the latest version of VDT.py.
Once downloaded, place the script in a convenient directory on your vCenter Server.
Easy as pie, right?
Initial Setup and Configurations
Now that you have the script in place, let’s configure it for your environment:
Open your SSH terminal prompt.
Navigate to the directory where VDT.py is located.
Run the script by typing python VDT.py.
Follow the onscreen prompts to set initial configurations such as server details and credentials.
VDT will now start performing its diagnostic checks. You’ll receive a detailed report highlighting any issues or misconfigurations. If all looks good, you’re ready for the upgrades.
In conclusion, VDT is more than just a script. It’s a safety net that ensures your VMware environments are robust before any major changes. Whether you’re a seasoned administrator or a newbie in the VMware world, incorporating VDT into your routine can save you from many potential headaches.
Preparing for vCenter Server Upgrades
Importance of Pre-Upgrade Checks
When it comes to upgrading your VMware vCenter Server, doing pre-upgrade checks is absolutely crucial. Think of it as prepping for a big journey; you wouldn’t set off without making sure your car had oil and gas, right? The same goes for your server! Skimping on these checks can lead to catastrophic issues.
So, what needs to be checked? Glad you asked! Here are a few essentials:
Passwords: Are all your admin and service accounts up-to-date?
Certificates: Do you have valid SSL certificates?
Storage: Is there enough space for the upgrade?
Network Configuration: Are all your settings correctly aligned?
Components to Verify Before Upgrading
Now that you know the importance of pre-upgrade checks, let’s delve into the components you need to verify:
VMware vCenter Server Version: Make sure it’s on version 8.0.3a.
Compatibility: Ensure all plugins and extensions are compatible with the new update.
Hardware Support: Double-check if your current hardware meets the new version’s requirements.
To give you a quick glance on what you’re working with:
vCenter Server Version
Updates
8.0.3a
Recent Broadcom Updates: Sept 17th
Common Pitfalls and How to Avoid Them
Ever walked right into a trap? Okay, maybe not literally, but missing these common pitfalls can feel just as bad:
Insufficient Storage: Always check you’ve got enough storage before hitting the ‘Upgrade’ button.
Ignoring Compatibility: Double-check your third-party tools and plugins for compatibility issues.
Skipping Backups: Never, ever skip backing up your current state. Trust me, you’ll thank yourself later!
Addressing these issues? Simple. Make backups a standard practice and audit your entire system before upgrading.
Tools and Resources for Smooth Upgrades
I get it, all this can seem overwhelming. But don’t worry, we have tools and resources to make this easier:
VDT (vSphere Diagnostic Tool): Handy script that checks passwords, certificates, and storage.
And here’s an example of a checklist to help you get started:
Break down your tasks, check them off one by one, and minimize headache later. Pre-checks, verifications, and backups are like setting a solid foundation before you build a house. Neglect them, and you risk everything crumbling down.
Stay tuned for more insights and tips about VMware vCenter Server upgrades. And remember, always be prepared!
RDU: Reducing Downtime During Upgrades
What is RDU and Why It’s Important
When we talk about RDU, we are referring to Reduced Downtime Upgrade. It’s a technique aimed at minimizing the service interruptions during software upgrades. Think of it like this: Imagine you’re a chef preparing a grand feast. The kitchen is bustling, but you need to replace some essential appliances. Wouldn’t it be ideal if you could swap them out without halting your cooking? That’s the essence of RDU. It’s crucial because in today’s fast-paced world, even a few minutes of downtime can lead to significant losses.
Steps to Implement RDU Effectively
So, how do you effectively implement RDU? Here are a few steps to guide you through:
Assess Your System: Start by evaluating what areas are critical. Identify the key components that absolutely cannot go offline.
Patch Testing: Test patches in a isolated environment first. This helps ensure compatibility and performance with your existing system.
Schedule Upgrades: Plan your upgrades during off-peak hours to minimize disruption. For global operations, late-night hours or early mornings are usually best.
Communication: Inform your team or customers about the upgrade schedule. Transparency can reduce frustration and support readiness.
Monitor and Validate: Post-upgrade, keep an eye on system performance. Validate that all components are working as intended.
Following these steps can significantly reduce the risks associated with system upgrades. Broadcom, for instance, has published a fix and workaround for the latest issues, demonstrating the importance of ongoing monitoring and timely updates.
Case Studies and Success Stories
Several organizations have successfully implemented RDU. Take the case of a mid-sized e-commerce company that faced significant downtime issues during their peak holiday season. By adopting RDU strategies, they managed to cut downtime by over 70%, which significantly boosted their sales and customer satisfaction. Another example is a financial institution that used RDU to perform critical updates without interrupting their essential transactional services.
Upcoming Patches and Updates to Watch For
Staying updated with the latest patches and updates is vital. The next major patch release to watch for is on September 17th. This patch, tagged as VMSA-2024-0019, addresses critical vulnerabilities ranging from CVSS 7.5 to 9.8. According to the report:
Given the severity, it’s crucial to act promptly. You may need to patch these vulnerabilities within seven days or even immediately, depending on your organizational policies. Here’s a summary of the upcoming patch details:
Patch Release Date
Patch Version
Critical Vulnerabilities
Sept 17th
VMSA-2024-0019
CVSS 7.5 to 9.8
Conclusion
Note: This is just a section of the blog, and other sections, such as the conclusion, would follow in subsequent sections. Stay tuned for more in-depth information.
Remember, staying proactive with updates can save your organization from critical vulnerabilities and potential downtimes. So, watch out for those patches and keep your systems robust!
Addressing Security Vulnerabilities
Overview of the Latest Security Patches
Security is an ongoing battle. Recently, Broadcom released significant patches addressing vulnerabilities CVE-2024-38812 and CVE-2024-38813. These patches are crucial for system security. Now, let’s look into what these patches are all about.
Published on September 17th, they target critical issues identified within certain high CVSS scores. The term “CVSS” stands for Common Vulnerability Scoring System, but what does it mean? And why are these scores significant? Let’s dive into that next.
Understanding CVSS Scores and Their Impact
CVSS scores range from 7.5 to 9.8, with anything above seven considered high priority. The CVSS score helps prioritize the urgency of applying security patches. A higher score means a more dangerous vulnerability.
Why care about these scores, you ask? Well, let’s put it this way: Imagine leaving your front door unlocked in a high-crime neighborhood. You’re asking for trouble. The same goes for your digital infrastructure. Ignoring these scores is like leaving the door wide open for cybercriminals.
“The severity of these are actually critical, and they the CVSS ranges from seven point five to nine point eight.”
CVSS Range
Critical Score
Patch Date
7.5 to 9.8
9.8
Sept 17th
CVSS Score Distribution
How to Apply These Patches
Applying patches isn’t rocket science, but it does need careful attention. You can’t just “set it and forget it.” Here’s a step-by-step guide:
Backup: Always backup your system before applying patches. It avoids any hiccups if things go south.
Download: Get the patch from authorized sources. Unofficial downloads can be even more dangerous.
Install: Follow the installation instructions carefully. Each patch has its unique procedure.
Verify: Once installed, check if the patch was applied successfully. No assumptions here.
One interesting thing to note is that people reported issues even after applying these patches. Broadcom has already provided fixes and workarounds. We’ve seen this happen within just a few days post-release.
Monitoring and Maintaining System Security
Okay, so you’ve applied the patches. Congratulations! But the job isn’t done yet. Continuous monitoring is essential to keep your system secure.
Monitoring involves:
Regularly scanning for new vulnerabilities and apply patches as needed.
Maintaining logs to track any unusual activities. Even the tiniest anomaly can be critical.
Hosting regular security audits to ensure everything is buttoned up.
Why is this crucial? Imagine you’re the captain of a ship. Wouldn’t you want to know if there’s a small leak? Regular checks make sure your ship doesn’t sink.
In the end, keeping an eye on security is like a healthy diet. It’s not a one-time thing. It’s a lifestyle choice. These practices make life difficult for cybercriminals.
So, folks, stay diligent, stay secure. As we continue our journey through addressing security vulnerabilities, I’ll be sharing more insights. For now, make sure you’re patched, monitored, and ready for anything.
Tips for a Smooth VMware Experience
Best Practices for Using VMware Tools
VMware is powerful, but to get the best out of it, you need to follow some crucial practices. Always keep VMware tools updated to ensure compatibility and performance. They come with essential drivers and daemons that enhance your virtual machine’s efficiency. Also, regular backups are non-negotiable. Imagine losing critical data just because of a skipped backup—yikes!
Another best practice is optimizing resource allocation. Allocate only what is necessary for your VMs. Over-allocation can lead to wasted resources and under-performance elsewhere. Also, remember the importance of DNS and NTP. A properly configured DNS ensures that your VMs communicate seamlessly, while NTP keeps them in sync. The importance of time synchronization can’t be overstated.
Real-life Scenarios and Troubleshooting Tips
I’ve encountered instances where a slight misconfiguration caused significant issues. For example, ignoring time synchronization can lead to a harsh time drift. In one case, we experienced a 4-minute delay that caused email alerts to be incorrectly timed. When we switched to a controlled time server, those issues vanished.
“I think somebody had actually built a Raspberry Pi time server for twelve dollars.”
Using a Raspberry Pi as a time server is cost-effective and gives you control over time synchronization. If you rely on public NTP servers, you could face unexpected downtimes or inaccuracies. Raspberry Pis can provide a stable and affordable alternative, especially for lab setups. Here’s a quick look at the costs involved:
Time Drift
Raspberry Pi Time Server Cost
4 minutes delay
$12
Common Mistakes and How to Avoid Them
Avoid the following common mistakes to keep your VMware experience smooth:
Skipping updates: Always update your VMware tools.
Ignoring DNS and NTP configuration: Make sure your VMs are dialed in.
Using public NTP servers: They are fine for labs but impractical for serious deployments.
Public time servers can seem convenient, but they’re not always reliable. If you rely on them, you’re dependent on external factors and networks. Set up your own time server, preferably using a simple Raspberry Pi, for consistency and control.
Future Trends and What to Expect in VMware Updates
VMware constantly evolves. Keep an eye out for improvements in automation and AI integration. These advancements will likely reduce manual configurations and optimize performance even further. The use of AI for predictive analysis and maintenance can potentially save you from unforeseen downtimes.
Imagine a fancier script that ensures your certificates and clusters are all in order by just running a command. VMware updates aim to simplify and enhance user experience continually.
Lab Setup Recommendations
A well-setup lab can significantly improve your VMware experience. Ensure you have a stable DNS and NTP configuration, and don’t hesitate to use Raspberry Pi time servers. They’re cheap, effective, and easy to manage. Here are a few key recommendations:
Dedicate a couple of Raspberry Pis for time synchronization.
Keep your DNS and network infrastructure robust.
Regularly update and test your configurations.
Common Time Management Issues
Here’s a visual representation of common time management issues faced in VMware:
TL;DR
Use updated VMware tools, optimize resource allocation, configure DNS and NTP, and consider using Raspberry Pis for time servers. Avoid common mistakes and watch out for future trends focused on automation and AI. Set up a stable lab and ensure time management issues are minimized for a smoother VMware experience.
In your #homelab, if you’re experimenting with VMware vSphere vSAN, having one or two 10GbE switches is essential for optimal performance. A cost-effective option is the RackSwitch G8124E, often available on eBay. Originally produced by Blade, this switch was later acquired by IBM and eventually became part of Lenovo’s networking portfolio.
BLADE Rackswitch G8124
If you own a Blade, IBM, or Lenovo RackSwitch G8124 10GbE network switch and encounter the frustrating “UNAPPROVED – SR SFP+ is DISABLED” message when using non-Blade, IBM, or Lenovo SFP+ modules, you’re likely searching for a way to bypass this issue. You’re in the right place!
I have no idea why Lenovo is so precious about this information and keeps it a secret. Officially, Lenovo didn’t want people tinkering with the maintenance prompt for fear of causing damage and unnecessary service calls. In reality, there’s a lot of money to be made in optics/modules, so for most vendors, including Lenovo, that’s likely the main reason they’ve kept this information private.
Personally, I do not understand why Lenovo is so secretive. The switches are currently available as scrap value on eBay, and Lenovo does not receive any money from the sale of these switches. These devices are out of warranty and will not be used in production, and Lenovo hasn’t sold these switches for over 14 years. Other compatible SFP+ modules are available that work with this SFP+ override, but based on my experience, the effort was a waste of time since modern SFP+ modules are not compatible with older switches from 2010!
Additionally, I uploaded photos of Blade SFP+ modules available on eBay for as little as £1 each to the Lenovo Forums, which resolved my issue without needing the SFP+ override mentioned above. However, as a result of trying to help others in the community by suggesting these purchase options, I was permanently banned from the Lenovo Forums! It’s frustrating when the goal is to assist fellow users, yet vendor lock keeps people from finding practical solutions. I’m not sure if they thought I was selling SFP+ modules! I didn’t post any direct links to eBay!
While this method can unlock non-vendor SFP+ modules, it’s crucial to note that your experience may vary depending on the optics used. We do not recommend using dual-rate optics with these older switches, as they may not offer optimal compatibility or performance. Always ensure the optics meet the required specifications and thoroughly test them in a controlled environment before deploying in live networks to avoid potential hardware or network disruptions.
Procedure to Unlock Non-Vendor SFP+ Modules:
For early 8.X code (8.3.X and lower) or older firmware versions (7.X, 6.X, 5.X), the following steps must be done through the serial console port (mini-USB, RS232 signaling). You need to have a Serial RS232 cable with a Mini-USB connector on the end; please note this is NOT USB!
Reload the switch (this will interrupt traffic).
Watch the serial console (9600, N, 8, 1). During the memory test (indicated by dots on the display), repeatedly press “Shift + M” until you see the “=>” prompt.
At the “=>” prompt, input these commands exactly as shown (case-sensitive):
setenvsfp Override
saveenv
printenv
boot OR reset OR reload
Command Breakdown:
setenv sfp Override: Sets the environment variable “sfp” to “Override”.
saveenv: Saves the override setting to NVRAM.
printenv: Verifies the variable has been set; confirm you see “sfp=Override”.
boot, reset, or reload: Restarts the switch with the new environment variable.
Important Notes:
This is primarily for lab use or urgent situations where a valid transceiver is unavailable. In production environments, always use IBM or Lenovo-branded optics to avoid issues.
After software upgrades, the variable may reset, so keep these instructions handy to reapply the override as needed.