With VMware vSphere 8.x out and vSphere Lifecycle manager making the shift from individual baselines to cluster images, there are some additional encounters you may have when integrating with our solutions from VMware or even other vendors.
I encountered an error recently in NSX 4.1.0.2.0.21761693 during host preperation I received the following error.
When clicking on the error for details and steps, you see
Go to the VMware Cluster >> Updates >> Image
You can perform an Image Compliance check manually or you will find there is my problematic host not showing compliant because it is missing NSX vibs
Click ‘Remediate All’, review your remediation settings and click ‘Remediate’. Once Remediation completes, I decided to reboot the host and once it came back up, inside NSX Manager I located the node and to the far right clicked on ‘View Details’ and click ‘Resolve’ to the prompt.
Monitor the installation status
This completed successfully, as the host now shows as prepared and ‘Success’
I’ve deployed 3 NSX Managers individually from the NSX OVA onto a single vCenter. By having 3 individual Managers, I have the option to create multiple clusters from each one (probably excessive and incorrect in my case). Instead my goal is to join all 3 individual managers to form a 3-node cluster and then assign a VIP.
My 3 NSX managers I will be referencing and joining are nsxcon1, nsxcon2 & nsxcon3
Here is an example of nsxcon1 UI reviewing the ‘Appliances’ section, you can see there is only a single appliance and an additional one cannot be added until a ‘Compute Manager’ (such as a vCenter) can be added.
I did verify CLI connectivity to each of the appliances by running
get cluster status
This command will return cluster health for the NSX Manager and any appliances that are part of the cluster, for this example, it’s only a single appliance
From the first NSX controller you will want to obtain the thumbprint by
get certificate api thumbprint
That will provide you the thumbprint of the targeted appliance
Moving onto the other node (nsxcon2) which we want to join to nsxcon1, we will use the following command
Here is an example of what it looks like when populated in that command and ran from the node we want to join to our primary one.
*Please ensure you have taken appropriate backups as this will take this node and try and join it to another cluster, being this should be a vanilla install, should not be too much to have to re-deploy.
After a couple of minutes we do receive the following prompt
We can then go back to nsxcon1 and verify with ‘ get cluster status’ and see that the cluster status is ‘DEGRADED’ however this is normal while the node is completing it’s process with joining and updating the embedded database.
We can take our ‘join’ command earlier we used on nsxcon2 and then run it on nsxcon3 again.
After running it, going back to nsxcon1 and checking cluster status..we now have 3 appearing
After a few minutes, our GUI has been fully populated with all NSX Managers reporting as stable
As a cherry on top, we will click on ‘Set Virtual IP’ and assign a dedicated IP address which also has it’s down DNS record.
There is our new virtual IP which has been assigned to one of the nodes
In preparation for vSphere 8 upgrades, I’m in the process of upgrading many of the solutions in the homelab before upgrading to the big 8.
I’m currently running NSX-T 3.2.1 with NSX Manager appliances. I have NSX deployed out to a cluster with a couple of Edge appliances in a cluster configuration.
For those that might’ve missed the word, it was announced early-2022 that NSX-T 3.x would be no longer and that the naming would be shifting to NSX with versions 4.x going forward. You can read more about this here.
The first step was to ensure I have a recent ‘Successful’ backup from within NSX-T manager itself.
When you go out to Customer Connect Downloads, you will want to download the NSX 4.0.1.1 *.mub Upgrade file.
Once the file is downloaded, I chose to upload it from my local system where I was using my browser to access the NSX interface.
Once the file uploads, the next step will be to click ‘Prepare for Upgrade’
The following process will take some time, you might even be prompted with a session timeout, In my instance I came up with the error “Repository synchronization operation was interrupted. Please click on resolve to retry. Repository synchronization failed. a ‘Retry’ ran and it completed the check successfully.
Once this process completed, it took me to step 2, the manager console will reload.
Click on the drop-down and select ‘All Pre-Checks’
After reviewing the results of the Upgrade, I reviewed the alarms and felt I wanted to move forward with the upgrade. The Edges are alarming due to Memory Consumption and the Manager alarms were relating to the NSX ‘audit’ account.
I selected to run the upgrade in ‘Serial’ and selected ‘After each group completes’ for Pause upgrade condition. click ‘Start’
The Edge upgrades completed successfully, I will click Next for the Hosts. There is also an option to ‘Run Post Checks’
The post checks ran fine and the next step is start the Hosts for upgrades.
The hosts upgrade completed successfully, I even ran the post upgrade check and it succeeded. The only gotcha for my cluster was I had to manually move some VMs onto other hosts and power some VMs down to conserve on resources, so once that was done, hosts entered maintenance mode.
The final step which was to upgrade the NSX Managers, click ‘Start’
So the upgrade failed immediately as the ‘audit’ account came back to bite me, there was a strange behaviour where when I update the password, the account was still showing a Password Expired status, I ‘Deactivated’ and ‘Activated’ the account and it showed Active. Once that was completed, the message also stated to take action on the ‘Alarm’ in NSX, so I went back and Acknowledged and Resolved alarms and did not leave any in Open status.
The upgrade will allow you to continue once you navigate back to System>>Upgrade. Go back to Step 1 and run the “Pre Check” for NSX Manager Only, before proceeding to the final upgrade step.
The upgrade completed successfully. You will notice the top left-corner banner will now only read ‘NSX’.
Running VMware NSX-T 3.2.1 on a clean install, no previous upgrade. Home lab is not sitting around doing much other than whatever idle tasks might be going on with the appliance. I decided to log in and BAM! The system section was not loading
Error code: 513002
Under System>>Configuration the following error would appear
I was able to navigate to check the basic health of my managers, and checked ‘View Details’ for each one; everything was up, all green, and space utilization looked good.
From the VM console logged in as admin, I tried ‘restart services controller ‘and no success.
While troubleshooting and looking for KBs and forums on the web, when running ‘get managers’ from admin console, one of my managers was in Standby, but then there was some intermittent activity where all went into Standby for a brief moment.
Not just this happening but my Appliances GUI was also now switching between ‘Healthy’ & Degraded.
Next step was to reboot one appliance at a time, which did not seem to fix the issue.
After taking a break and happen to be doing something else in the home lab, I had to shut down and restart the vCenter, well I happen to check later on in the day and NSX was able to load with no problems.
I’m going to keep this up in the event the issue comes around, I plan on updating.
