Skip to main content

RAM Disk full on ESXi host



Sometime we used to get issue where were are not able to perform vMotion or logging was unable to write under /var/log.. While trying to do some normal things – like vMotion. I noticed an error which states just “A general system error occurred.”  On further investigation, I found that the underlying message was an out of disk space message while trying to proceed with a Storage vMotion.

Observations during issue

While vMotion – “A general system error occurred:”
While performing Storage vMotion – “/var/log/vmware/journal/xxxx error writing file. There is no space left on the device.”

Steps during troubleshooting

  • Go to Configuration tab on host in vCenter client, go to Security Profile, click Properties link on the Services section.
  • Scroll down to SSH and highlight – click options – click start to start SSH service.
  • Use putty or reflections to ssh to the host.
  • If you get a connection rejected – root filesystem ramdisk is probably full.
  • Go to console (either through KVM or OA for blades)
  • F2 to login, login, arrow down to Troubleshooting Options, select Enable ESXi Shell.
  • Press ALT-F1 to change to management shell and login (same root credentials).
  • Run ‘vdf -h’ and look for root filesystem – should look like:
    Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        3M       28M  10% --
  • If it is 0M available and 100% used, that’s the problem.  Try to clear up space:
    cd /var/log/
 ls -la
  • Check size of the hpHelper.log file – likely pretty large.  Reset the file, if large.
    > hpHelper.log
After clearing the hpHelper.log, were were able to bring the host into normal state and all task were observed as normal. Identified issue detected with the HP agents inside the custom image of ESXi provided by HP  It seems in some circumstances that the hpHelper.log file can become very large, filling the RAMdisk and causing the issues.  Its a first for me and I have not observed the issues on any of my other ESXi hosts running on Proliant rack-mount or blade servers.  
Happy Sharing... :)
Labels:

Comments

Post a Comment

Popular posts from this blog

Changing the FQDN of the vCenter appliance (VCSA)

This article states how to change the system name or the FQDN of the vCenter appliance 6.x You may not find any way to change the FQDN from the vCenter GUI either from VAMI page of from webclient as the option to change the hostname always be greyed out. Now the option left is from the command line of VCSA appliance. Below steps will make it possible to change the FQDN of the VCSA from the command line. Access the VCSA from console or from Putty session. Login with root permission Use above command in the command prompt of VCSA : /opt/vmware/share/vami/vami_config_net Opt for option 3 (Hostname) Change the hostname to new name Reboot the VCSA appliance.   After reboot you will be successfully manage to change the FQDN of the VCSA . Note: Above step is unsupported by VMware and may impact your SSL certificate and face problem while logging to vSphere Web Client. If you are using self-signed certificate, you can regenerate the certificate with the
2 comments
Read more

Collecting Logs from NSX-T Edge nodes using CLI

  This article explains how to extract the logs from NSX-T Edge nodes from CLI. Let's view the steps involved: 1) Login to NSX-T  Edge node using CLI from admin credentials. 2) Use of  " get support-bundle " for Log extraction. get support-bundle command will extract the complete logs from NSX-T manager/Edge nodes. nsx-manager-1> get support-bundle file support-bundle.tgz 3) Last step is to us e of " copy file support-bundle.tgz url " command. copy file will forward your collected logs from the NSX-T manager to the destination(URL) host from where you can download the logs. copy file support.bundle.tgz url scp://root@192.168.11.15/tmp Here, the URL specified is the ESXi host ( 192.168.11.15) under /tmp partition where logs will be copied and from there one can extract it for further log review. Happy Learning.  :)
Post a Comment
Read more

Removing NSX-T manager extension from vCenter

In NSX-T starting from ver 2.4 NSX-T appliance got decoupled from vCenter where now its not mandatory to run NSX-T on vCenter platform only. Now NSX-T can be managed through standalone ESXi host, KVM or through container platform. As in version 2.4 there is still an option available to connect vCenter to NSX-T using Compute Manager. Here in this blog we will learn how we can unregister and register NSX-T extenstion from vCenter in case of any sync or vCenter connectivity issue with NSX-T. Lets get started.. 1) Login to NSX-T UI Go to -> System ->Compute Manager Here, vCenter is showing in Down status where the status is showing as "Not Registered" 2) When we click on "Not Registered" option its states below error. 3) When try to click on Resolve option its states below. At this stage if the Resolve option doesn't work then its require the remove the NSX-T extenstion from vCenter. To remove the NSX-T e
Post a Comment
Read more