Looking back on 2017

Posted on December 31, 2017 by

Just want to register my memory on 2017 , It has been wonderful journey with lot of happiness and also some bad memories .

So after doing almost six years of consultant job , I joined as the full time employee around mid 2016  and it was totally different  experience .When I was doing the consultant job , had lot of time for outside activity other than my daily responsibilities, like preparing and learning new stuff to improve my skills ,writing blog but after joining full time job the story is totally different and in my personal life my kids also now needed lot of time from me. On all these changes, I somehow managed to post at least one blog on each month.

2017 I have done some significant project like designing DR and Implemented the VMware SRM  and worked in the project to setup internal cloud in our environment. This year VMworld was so informative and had chance to meet some top personalities in the market. Year-end was not so good because of the recent layoff which affected lot of my close friends in the office. It is really nightmare for everyone and now lot of pressure to project our-self to show that we are worth for the company growth. Even though the vision of the company is to move to external cloud but one positive note is company are still showing interest on internal infrastructure  so the goal on 2018 is to learn open stack and AWS to improve the skills. I know 2018 going to be very challenging because of VISA issues in US and the new vision of our company. It will be very interesting and as i said challenging  year to prepare myself to respond for the situation in up-coming months..

 

Wishing everyone a very happy new year and hope will spend more time on writing the blogs with helpful information ..

Advertisements
Posted in year 2017 | Tagged | Leave a comment

VCSA 6.5 upgrade\Installation issues and fixes

Posted on November 30, 2017 by

Recently I have started upgrading the VC6.0 U3 ( VCSA ) to 6.5 U1 and encountered few issues  so  same like by previous blogs on VC6.0  issues , planned to do it for VC6.5 also and  find the old blogs below.

vCSA 6.0 Installation issues 

vmware-kb-articles-with-no-resolution

We have around 25 different vCenters to be upgraded and will update the blog accordingly .

Issue 1.

Invalid Appliance (OS) root password

So while upgrading the appliance we need to provide the SSO Password and also source VC\PSC appliance root password and it was failing with the OS root password is invalid. I have verified  the password which is correct and later found that SSO and OS password is same which fails to accept . Once I have changed the OS password, issue got fixed.

Issue 2.

Unable to retrieve the migration assistant extension on source vCenter server.

So as per the KB 2148400 , environment which is not using the VUM , we need to remove the com.vmware.vcIntegrity extension. KB 1025360 will guide the steps.

Issue 3.

The Installer is unable to connect to the vCenter Appliance Management Interface.

So my source VC was running in one cluster and trying to implement the upgrade 6.5 appliance on different cluster . Since the network is different it got failed on the end of the Stage 1 . Once I pointed the upgrade 6.5 appliance to the same cluster issue got fixed.

Issue 4.

Could not reach the given source vCenter Server address on port 22

Lot of blogs are pointed to check the SSH on the source appliance but in my case it was already enabled so I just restarted all the services before the upgrade and the issue got fixed on the next try.

 

 

 

Posted in ESXi issue, VC6.0 Appliance Installation Issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , | Leave a comment

Options to check and alert the vcenter certificate expiration

Posted on October 9, 2017 by

Last week one of our vCenter went down because of the machine certificate got expired and it took some time to find out the issue so I thought it will be helpful to show the options to verify the certificate and make sure to enable the alarm.

Since the certificate as expired most of the services will fail to work properly since it cannot function/use the certificate it is assigned to use.

In our case, we are unable to vMotion because the service to vMotion (vmware-sps) is unable to connect to vpxd due to “server certificate chain not verified.”

Below is the log path to verify .

/var/log/vmware/vmware-sps/sps.log

com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Below is the command to verify the Machine certificate.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list –store MACHINE_SSL_CERT –text |less

Solution certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry list –store machine –text |less

Also we can check the same using the web-browser  .

Using the “ /usr/lib/vmware-vmca/bin/certificate-manager” Replace the certificates on the vCenter via option 3 (just the MACHINE_SSL) or if it is with internal CA then follow the steps here.

So to make the alarm configured for the certificate expiration, already by default 30 days threshold is configured in the vcenter and You can change how soon you are warned with the vpxd.cert.threshold advanced option.

  1. Log in to the vSphere Web Client.
  2. Select the vCenter Server object, the select the Manage tab and the Settings subtab.
  3. Click Advanced Settings, select Edit, and filter for threshold.
  4. Change the setting of vpxd.cert.threshold to the desired value and click OK.

Also make sure under Alarm settings – Certificate Status – Enable this alarm is active so that according to the threshold  we will get the alarm notification when the issue occurred.

Reference :

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-D3DB7279-0A25-4AA8-83A0-F34E5676A8B9.html

Posted in Certificate, ESX command, Replacing vCenter 6.0 SSL Certificate, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware, vSphere 6.0 Template. | Tagged , , , | Leave a comment

Bug in NFS mount while using SRM

Posted on September 30, 2017 by

We were configuring the SRM with Tintri storage and while configuring the Array pairing in storage replication , SRM was failed to mount the Tintri datastore.

So as per Tintri we need to create the service group in the production site and it will automatically create the folder in the destination for the replication. Once the storage part is done, we started configuring the Array Pair in the SRM and it got completed without any error but in the below array pair column it was showing only the service group without the datasotre mapping.By the result of this we cant create the protection group which need datastore  groups mapping.

SRM

By the help of Tintri support we have tried lot of options like deleting the service group and re-creating it but nothing helped. At last found it is known  SRM issue with NFS datastore having mounted in the ESXi using the FQDN. ( KB 2009622 )

Site Recovery Manger cannot match the Fully Qualified Domain Name (FQDN) or short name of the NFS server that is used to mount the source NFS volumes on the ESX/ESXi hosts with the StoragePort information that is reported in the SRA response to the discoverdevices command.

Finally we remounted all the datastore with IP address and the issue got resolved

 

Reference

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009622

Posted in SRM, VMware | Tagged , , , , | Leave a comment

Mind-Map for dvSwitch.

Posted on August 31, 2017 by

My flight got delayed after the VMworld and have to spend few hours in the airport so just to make something interesting made this Mind-map for dvSwitch . Hope it will be helpful for the certificate preparation.

Download

Posted in Vcenter Appliance, vCSA 6.0, VMware | Leave a comment

VMware support log size issue and workaround (VCSA ).

Posted on July 9, 2017 by

In our environment we have around 12 VCSA 6.0 U2 ( vCenter  Appliance)  in different region with various sizes , most of our environment will be running with around 20-25 ESXi hosts in two or three clusters and  800 – 1300 VMs so whenever any issue on the environment  it will be always challenging to upload the logs to the VMware support because the support log size will be more that 15 GB to 20 GB and at one point log generation will fail because of the log size partition and even after increasing the space of the log partition, it will be big challenge to upload it to the VMware FTP site .

VMware support engineers have no clue about the reason for the huge bundle log size and have no other option to fix the log to upload it in their FTP site  and in most cases we use to upload the specific  date and log type only to the VMware  for the troubleshooting.

I was searching lot of blogs and slack channels but didnt help much but came to know the plugin called  VMware support assistance tool which we can directly upload the logs from the Vcenter to the VMware portal so I decided to install the same.

One drawback of this plugin is if the web-client session is time-out then it will interrupt the log upload and the process will be closed so we need to increase the web-client session time-out and it can be done by the help of KB 2040626 so we have increased the session time-out from the default 30 mins to 6 hours which helped us to successfully upload the logs to the VMware portal .

But still it is the security concern on having the web-client session opened for 6 hours and also really want to find out the reason behind the bundle log size growth.After long wait and  several  conversation with VMware senior engineers , identified that under /storage/core  all the old vpxd.core  dump will be stored which is not required and we only required live_core.VPXD.* for the vcenter. As per the suggestion we have deleted all the old files form our lab VC and now the log bundle size reduced significantly .

Note: Pls have proper backup or snapshot before deleting .  If you wrongly delete the live_core then it will crash the vCenter.

Reference :

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2040626

http://www.virtuallyghetto.com/2012/09/configuring-vsphere-web-client-session.html

http://www.virtuallyghetto.com/2014/07/how-to-generate-specific-support-log-bundles-for-vcenter-esxi-using-vsphere-api.html

Posted in Vcenter Appliance, vCSA 6.0, VCSA6.5 | Tagged , , , | Leave a comment

Powercli script to validate the basic vSphere Hardening

Posted on June 18, 2017 by

Last month I was working in a security hardening project and implemented as per the standard recommended by the VMware.

We have to show to our internal security team that  all the vsphere environment is protected as per the VMware recommendation so we built small web portal which will call the powercli script in the background to validate the hosts and provide the result.

Script will validate the below settings and provide the results

ESXi.config-ntp and service status
ESXi.config-persistent-logs
ESXi.enable-ad-auth
ESXi.config-persistent-logs
ESXi.set-account-auto-unlock-time
ESXi.set-shell-interactive-timeout
ESXi.set-shell-timeout
DNS IP check
Allowing only corresponding IP in DNS firewall  for the UPD\TCP.

Result will show each tasks status and if it is fully protected then it will show as fully protected or else it will show which one is not as per the standard and in case if only one or two tasks are not up to the standard then it will show that corresponding tasks to change and rest it will show as 1 which means it is protected as per the recommendation .

Like mentioned results which is showing as 1 is good and if all the condition is 1 then it will be considered as true and host is fully protected.

Pls download the script from the below link.

Git-hub

 

Posted in PowerCLI, Powershell, VMware | Leave a comment