Tips to check VCSA sessions.

Posted on May 31, 2021 by Ganadmin

Below grep NumSessions shows the currently active sessions on the vpxd profiler. The longer query showing HttpSessionObject is checking for all HTTP sessions that were created over a certain period of time.

The HttpSessionObject notes a unique session at a point in time in the log. We filer on this because there are many different objects noted in the profiler log that don’t relate to HTTP sessions.

Login in to SSH – cd \var\log\vmware\vpxd\

grep NumSessions vpxd-profiler*.log | less

grep ClientIP vpxd-profiler*.log | grep HttpSessionObject | grep -v com.vmware | grep -v “”” | cut -d “‘” -f 3-6 | sort | uniq –count | sort -nr | less

Reference : https://www.youtube.com/watch?v=eFM_ewwy2ys&ab_channel=VMworld

Posted in vCSA 6.0, VCSA6.7, VMware | Tagged , , , , | Leave a comment

Issue on SSH login to the ESXi 6.7 host with the AD user account

Posted on April 30, 2021 by Ganadmin 
We were not able to login to the ESXi ssh using the AD account and when we tried to leave the account or add the domain it is getting failed.

[root@esx:~] /usr/lib/vmware/likewise/bin/domainjoin-cli join prd.com admin
 - While adding the host got below error:
 
Error: LW_ERROR_LDAP_CONSTRAINT_VIOLATION 


Deleted the stale entry/ESXi computer account from Active Directory.

Post deleting the account, ESXi was successfully able to leave the domain. Used below command to leave the domain:
 
[root@esx:~] /usr/lib/vmware/likewise/bin/domainjoin-cli leave
- Used below command to add the ESXi back to the domain which was successful. 
 
[root@esx:~] /usr/lib/vmware/likewise/bin/domainjoin-cli join prd.com admin
Joining to AD Domain:   prd.com
With Computer DNS Name: esx.prd.com
 SUCCESS
- Post joining the ESXi to domain team was successfully able to login to ESXi host using domain user account.

Posted in ESXi issue, VMware | Tagged , , | Leave a comment

Tip to check the ESXi\vCenter errors using the Splunk.

Posted on March 31, 2021 by Ganadmin

Recently we had “All path down issue” in one of our host and I was looking to find how many events and how long this issue was there in host and identified the below steps in the Splunk in which we can highlight the key word to find the list.We can easily get the details from the ESXi but I felt below steps will be useful for the other use cases.

Make sure we have the Add-on for VMware https://splunkbase.splunk.com/app/3215/ in splunk which is no cost and it will install the VMware sourcetype parsers.

1. Click on Event Action > Extract Fields to start the wizard

2. Select Regular Expression > highlight to select a value > name the field > continue on to validation and complete the wizard.

When you click the events it will show all the events regarding the word you highlighted .

Useful Links:

https://splunkbase.splunk.com/app/3975/

Posted in logs, vCSA 6.0, VCSA6.5, VCSA6.7, VMware | Tagged , | Leave a comment

Bug noticed on VCSA 14367737 Syslog configuration.

Posted on February 28, 2021 by Ganadmin

We are running the VCSA 14367737 and it can’t be upgraded because we have the internal cloud stack on top of the vCenter and it supports only the VCSA version 14367737. I have tried forwarding the VCSA logs to the Syslog server ( SPLUNK ) and noticed after the configuration it worked for few hours and stopped working and we have to restart the service manually systemctl restart rsyslog to forward the logs again to the Splunk server.

After trying few options and in our test environment we have tried upgrading the VC to different version and noticed the issue got fixed on the vCenter Appliance 6.7 Update 3g (6.7.0.44000) 16046470.Eventough in the release notes they havent mentioned anything on this issue , it looks like they have upgraded the rsyslog version on ths VCSA version.

As the workaround we can configure the cron and restart the service for every two hours.

Posted in VCSA6.7, VMware | Tagged , , | 1 Comment

Packet drop issue on HP Gen 9 \ Gen 10 servers running ESXi6.7.

Posted on January 31, 2021 by Ganadmin

We have noticed the packet drop on all of our HP BL460c Gen 9 \ Gen 10 across the region which is running ESXi, 6.7.0, 16316930 and the Network adapter presently installed on the server is HPE FlexFabric 10Gb 2-port 536FLB Adapter which is Qlogic Adapter.

Version which comes with the HP custom image includes the qfle3  driver version 1.1.6.0-1OEM.650.0.0.4598673 and We have tried updating the driver \ firmware of the HP Enclosure \ OA \ Virtual Connect to the below versions but didn’t fix the issue.

OA Firmware : 4.96

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_8e583ffa28874a53aa272b959b

      3.. Upgrade the Virtual connect firmware on one switch and another switch.

HP Virtual Connect Firmware: 4.85

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_f99f0bc5bfc4414aac021f81af#tab3

Solution:

After a lot of options tried, HP has recommended installing the below driver version and packet drop issue has been fixed.

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_fca9a16a601345919247b0c240#tab-history.

[root@esx106:~] esxcli software vib install -v “/cp039955/QLogic-Network-iSCSI-FCoE-v2.0.102-14793946/QLogic-Network-iSCSI-FCoE-v2.0.102-offline_bundle-14793946/vib20/qfle3/QLC_bootbank_qfle3_1.0.87.0-1OEM.670.0.0.8169922.vib”

Installation Result

   Message: Host is not changed. Reboot is pending from previous transaction.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed:

   VIBs Skipped: QLC_bootbank_qfle3_1.0.87.0-1OEM.670.0.0.8169922

I think as per the ESXi Patch advisory it is mentioned QFLE3 is 1.0.50.11-9vmw.670.0.0.8169922 so we need to have something near to the version and once installed the driver recommended by HP which is 1.0.87.0-1OEM.670.0.0.8169922 fixed the issue for us.

Posted in ESXi issue, ESXi Patches, HP, VMware | Tagged , , , , , , , | 4 Comments

Memories of 2020

Posted on December 31, 2020 by Ganadmin

We have started migrating our Tier-1 infrastructure to AWS and most of our internal applications have been moved to the cloud and learned lot of new AWS services and new technologies.

Around SEP, I was requested to support our internal cloud team which is running cloud-stack with VMware and after a very big gap again back to VMware and virtualization technology. Initially it was very difficult for the change over but now very much back in to the track.

After moving to the internal cloud team , I got the opportunity to take care of the Scrum master role and started doing the same and planning to finish the certification.

Even tough because of the pandemic there were lot of challenges , I had a good 2020 in my professional life and looking forward for the new year 2021..

Posted in Uncategorized | Tagged , , | Leave a comment

Easy way to uninstall the Trend Deep Security agent.

Posted on November 30, 2020 by Ganadmin

I was searching the easy way to uninstall the Trend Agent on the windows 10 and find the below command useful.

Get-Package -Name  “Trend Micro Deep Security Agent” | Uninstall-Package

Or

msiexec.exe /x <exact MSI package name>.msi /quiet

Reference:

https://success.trendmicro.com/solution/1055096-performing-silent-uninstallation-of-deep-security-agent-dsa-from-windows-machine

Posted in Trend Micro Deep Security, Trend Micro Deep Security 9.5 ( Deep Security Agent ) | Leave a comment

AWS Compute related updates

Posted on October 31, 2020 by Ganadmin

AWS End of Support Migration Program for Windows Server now available as a self-serve solution for customers

Resource Access Manager Support is now available on AWS Outposts

New course for Amazon Elastic Kubernetes Service

Amazon EKS now supports Kubernetes version 1.18

AWS Lambda Extensions: a new way to integrate Lambda with operational tools

AWS Compute Optimizer enhances EC2 instance type recommendations with Amazon EBS metrics

Amazon EBS CSI driver now supports AWS Outposts

Amazon ElastiCache on Outposts is now available

AWS Elastic Beanstalk Adds Support for Running Multi-Container Applications on AL2 based Docker Platform

AWS Batch introduces tag-based access control

Amazon EC2 G4dn Bare Metal Instances with NVIDIA T4 Tensor Core GPUs, now available in 15 additional regions

AWS Launch Wizard now supports SAP HANA backups with AWS Backint Agent

Posted in AWS, EC2 | Tagged | Leave a comment

Steps to blacklist the problematic DCs in VMware VCSA 6.7U3

Posted on September 21, 2020 by Ganadmin

We had a DNS issue in one of the DC running active directory integrated  DNS service and it caused our vCenter to fail to connect the domain in AD so we have changed the  DNS to the IPs which is working properly but identified still AD authentication getting failed and in the VAR\LOG\Messages it was still pointing to the problematic DC and failing to authenticate.

After a few research got the instruction from the VCSA6.7 U3b release notes about the steps to blacklist the DCs and added the problematic DC IP as mentioned below.

Active Directory authentication or joining a domain is slow

Active Directory authentication or joining a domain might be slow when configured with Integrated Windows Authentication (IWA), because of infrastructure issues such as network latency and firewalls in some of the domain controllers.

This issue is resolved in this release. The fix provides the option to blacklist selected domain controllers in case of infrastructure issues.

To set the option, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs DC_IP1,DC_IP2,...
# /opt/likewise/bin/lwsm restart lwreg

To revert to the default settings, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs ""
# /opt/likewise/bin/lwsm restart lwreg

But still we noticed the VC is connecting to the problematic DC and also in the file /var/lib/likewise/krb5-affinity.conf it was showing the problamatic DC IP and when we tried to change it manually , automatically it got updated to the old problamatic DC IP .

After research we added the VC subnet in Active Directory Sites and Services to the new DCs and waited for few mins and noticed in the krb5-affinity.conf the new DC IPs got updated and issue got fixed by pointing the VC to the correct DC and ignoring the problematic DC.

Note : BlacklistDCs will work only from the 6.7U3b version.

Useful links:

https://kb.vmware.com/s/article/2127213

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.psc.doc/GUID-8C553435-27CD-4410-ACA9-9A84EA1D7334.html

https://kb.vmware.com/s/article/53698

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-vcenter-server-67u3b-release-notes.html

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VCSA6.7, VMware | Tagged , , , | Leave a comment

cfn-lint useful tool for the CloudFormation.

Posted on August 31, 2020 by Ganadmin

Recently had the chance to learn about the CFN-LINT tool which is a very useful tool to validate the CloudFormation template directly using the editor and it makes us create the CF without any error and secured.

https://github.com/aws-cloudformation/cfn-python-lint

Posted in AWS, AWS Server Migration Service, EC2 | Tagged , | Leave a comment