Some time back I had an issue where my NSX-T lab environment was showing unstable status. My environment consists of 3 NSX-T manager nodes aligned with the VIP IP address.
The issue where I was unable to access my NSX-T console through VIP IP address nor with my other NSX-T nodes. It's quite intermittent I was able to access console UI from one of the manager node using admin account. However, unable to login to the manager's node using SSH with admin or root account.
As I said its quite intermitted where I managed to access the manager UI console.
In the below Figure:1, it states that 1-2 manager nodes were showing unavailable.

set user <username> password <new password> old-password <old-password>
ie:
set user root password VMware1!VMware1!! old-password VMware1!VMware1!
As per the above Logrotate.conf snippets, It states that the Syslog rotation was owned by the Root user.
The log rotate runs as a CRON daily task, executed by the "root" user. Since the Root password had expired, the daily log rotation CRON job was failed to authenticate for the rotation.
The log rotation runs as a CRON daily task, executed by the ‘root’ user. Since the root password expired, the daily log rotation CRON job was failing to authenticate.
logrotate.conf
All the above logs state that the issue with unstable of NSX-T is due to /var/log full occupancy by 100% and that's because unable to rotate the syslog. As logrotate.conf for syslog was governed by root partition and in this case root partition was expired and confirm the root cause of this issue.
Auth.log
<87>1 0000-00-00T10:34:01.345432_00+00 nsxt000010.virtualvmx.com CRON 5324—pam_unix{cron:account_:exipred password for user root (password aged)
<87>1 0000-00-00T10:34:01.494949_00+00 nsxt000010.virtualvmx.com CRON 3423—pam_unix{cron:account_:exipred password for user root (password aged)
<87>1 0000-00-00T10:34:01.928345_00+00 nsxt000010.virtualvmx.com CRON 8765—pam_unix{cron:account_:exipred password for user root (password aged)
<87>1 0000-00-00T10:34:01.492823_00+00 nsxt000010.virtualvmx.com CRON 4323—pam_unix{cron:account_:exipred password for user root (password aged)
<87>1 0000-00-00T10:34:01.492384_00+00 nsxt000010.virtualvmx.com CRON 7665—pam_unix{cron:account_:exipred password for user root (password aged)
<87>1 0000-00-00T10:34:01.492838_00+00 nsxt000010.virtualvmx.com CRON 4827—pam_unix{cron:account_:exipred password for user root (password aged)
All the above logs state that the issue with unstable of NSX-T is due to /var/log full occupancy by 100% and that's because unable to rotate the syslog. As logrotate.conf for syslog was governed by root partition and in this case root partition was expired and confirm the root cause of this issue.
One can validate the root password in NSX-T using the below command.
get user <username> password-expiation
ie:
get user root password-expiration
So, Its quite important to validate the root password expiration to avoid this kind of scenario into your environment.
Comments
Post a Comment