Packet drop issue on HP Gen 9 \ Gen 10 servers running ESXi6.7.

Posted on January 31, 2021 by Ganadmin

We have noticed the packet drop on all of our HP BL460c Gen 9 \ Gen 10 across the region which is running ESXi, 6.7.0, 16316930 and the Network adapter presently installed on the server is HPE FlexFabric 10Gb 2-port 536FLB Adapter which is Qlogic Adapter.

Version which comes with the HP custom image includes the qfle3 driver version 1.1.6.0-1OEM.650.0.0.4598673 and We have tried updating the driver \ firmware of the HP Enclosure \ OA \ Virtual Connect to the below versions but didn’t fix the issue.

OA Firmware : 4.96

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_8e583ffa28874a53aa272b959b

3.. Upgrade the Virtual connect firmware on one switch and another switch.

HP Virtual Connect Firmware: 4.85

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_f99f0bc5bfc4414aac021f81af#tab3

Solution:

After a lot of options tried, HP has recommended installing the below driver version and packet drop issue has been fixed.

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_fca9a16a601345919247b0c240#tab-history.

[root@esx106:~] esxcli software vib install -v “/cp039955/QLogic-Network-iSCSI-FCoE-v2.0.102-14793946/QLogic-Network-iSCSI-FCoE-v2.0.102-offline_bundle-14793946/vib20/qfle3/QLC_bootbank_qfle3_1.0.87.0-1OEM.670.0.0.8169922.vib”

Installation Result

Message: Host is not changed. Reboot is pending from previous transaction.

Reboot Required: true

VIBs Installed:

VIBs Removed:

VIBs Skipped: QLC_bootbank_qfle3_1.0.87.0-1OEM.670.0.0.8169922

I think as per the ESXi Patch advisory it is mentioned QFLE3 is 1.0.50.11-9vmw.670.0.0.8169922 so we need to have something near to the version and once installed the driver recommended by HP which is 1.0.87.0-1OEM.670.0.0.8169922 fixed the issue for us.

Posted in ESXi issue, ESXi Patches, HP, VMware | Tagged ESXi Drivers, ESXi issue, ESXi Patches, HP, hp c7000 enclosure, HP Drivers, HP Firmware, HP Virtual Connect | 4 Comments

Memories of 2020

Posted on December 31, 2020 by Ganadmin

We have started migrating our Tier-1 infrastructure to AWS and most of our internal applications have been moved to the cloud and learned lot of new AWS services and new technologies.

Around SEP, I was requested to support our internal cloud team which is running cloud-stack with VMware and after a very big gap again back to VMware and virtualization technology. Initially it was very difficult for the change over but now very much back in to the track.

After moving to the internal cloud team , I got the opportunity to take care of the Scrum master role and started doing the same and planning to finish the certification.

Even tough because of the pandemic there were lot of challenges , I had a good 2020 in my professional life and looking forward for the new year 2021..

Posted in Uncategorized | Tagged 2020, 2021, newyear | Leave a comment

Easy way to uninstall the Trend Deep Security agent.

Posted on November 30, 2020 by Ganadmin

I was searching the easy way to uninstall the Trend Agent on the windows 10 and find the below command useful.

Get-Package -Name “Trend Micro Deep Security Agent” | Uninstall-Package

msiexec.exe /x <exact MSI package name>.msi /quiet

Reference:

https://success.trendmicro.com/solution/1055096-performing-silent-uninstallation-of-deep-security-agent-dsa-from-windows-machine

Posted in Trend Micro Deep Security, Trend Micro Deep Security 9.5 ( Deep Security Agent ) | Leave a comment

AWS Compute related updates

Posted on October 31, 2020 by Ganadmin

AWS End of Support Migration Program for Windows Server now available as a self-serve solution for customers

Resource Access Manager Support is now available on AWS Outposts

New course for Amazon Elastic Kubernetes Service

Amazon EKS now supports Kubernetes version 1.18

AWS Lambda Extensions: a new way to integrate Lambda with operational tools

AWS Compute Optimizer enhances EC2 instance type recommendations with Amazon EBS metrics

Amazon EBS CSI driver now supports AWS Outposts

Amazon ElastiCache on Outposts is now available

AWS Elastic Beanstalk Adds Support for Running Multi-Container Applications on AL2 based Docker Platform

AWS Batch introduces tag-based access control

Amazon EC2 G4dn Bare Metal Instances with NVIDIA T4 Tensor Core GPUs, now available in 15 additional regions

AWS Launch Wizard now supports SAP HANA backups with AWS Backint Agent

Posted in AWS, EC2 | Tagged AWS | Leave a comment

Steps to blacklist the problematic DCs in VMware VCSA 6.7U3

Posted on September 21, 2020 by Ganadmin

We had a DNS issue in one of the DC running active directory integrated DNS service and it caused our vCenter to fail to connect the domain in AD so we have changed the DNS to the IPs which is working properly but identified still AD authentication getting failed and in the VAR\LOG\Messages it was still pointing to the problematic DC and failing to authenticate.

After a few research got the instruction from the VCSA6.7 U3b release notes about the steps to blacklist the DCs and added the problematic DC IP as mentioned below.

Active Directory authentication or joining a domain is slow

Active Directory authentication or joining a domain might be slow when configured with Integrated Windows Authentication (IWA), because of infrastructure issues such as network latency and firewalls in some of the domain controllers.

This issue is resolved in this release. The fix provides the option to blacklist selected domain controllers in case of infrastructure issues.

To set the option, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs DC_IP1,DC_IP2,... # /opt/likewise/bin/lwsm restart lwreg

To revert to the default settings, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs "" # /opt/likewise/bin/lwsm restart lwreg

But still we noticed the VC is connecting to the problematic DC and also in the file /var/lib/likewise/krb5-affinity.conf it was showing the problamatic DC IP and when we tried to change it manually , automatically it got updated to the old problamatic DC IP .

After research we added the VC subnet in Active Directory Sites and Services to the new DCs and waited for few mins and noticed in the krb5-affinity.conf the new DC IPs got updated and issue got fixed by pointing the VC to the correct DC and ignoring the problematic DC.

Note : BlacklistDCs will work only from the 6.7U3b version.

Useful links:

https://kb.vmware.com/s/article/2127213

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.psc.doc/GUID-8C553435-27CD-4410-ACA9-9A84EA1D7334.html

https://kb.vmware.com/s/article/53698

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-vcenter-server-67u3b-release-notes.html

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VCSA6.7, VMware | Tagged joining PSC with AD, Vcenter Appliance AD authentication, VCSA, VCSA6.7 | Leave a comment

cfn-lint useful tool for the CloudFormation.

Posted on August 31, 2020 by Ganadmin

Recently had the chance to learn about the CFN-LINT tool which is a very useful tool to validate the CloudFormation template directly using the editor and it makes us create the CF without any error and secured.

https://github.com/aws-cloudformation/cfn-python-lint

Posted in AWS, AWS Server Migration Service, EC2 | Tagged AWS, cloudformation | Leave a comment

vCenter VPXD crashes because of high memory.

Posted on July 31, 2020 by Ganadmin

We have a vcenter environment with around 500 ESXi hosts running on multiple clusters and for the past several weeks we had the issue of Vcetner down because of VPXD crash and the service will be in stopped status.

VMware support identified the VCDB growth is huge with high CPU and memory usage on the vCenter and they started to investigate the same.

[Analysis]
Most of the memory usage is contributed by the Events

Signature 7f46978acb30 (Vmomi::KeyAnyValue) has 81430551 instances taking 0xd0aaf2b8(3,500,864,184) bytes.
Signature 562dc2a74a90 (Vmomi::Primitive<std::string>) has 55823822 instances taking 0x50096420(1,342,792,736) bytes.
Signature 7f4696eb77d0 (Vim::Event::ManagedEntityEventArgument) has 21574134 instances taking 0x37913be0(932,264,928) bytes.
Signature 7f4696eb7870 (Vim::Event::DatacenterEventArgument) has 13307524 instances taking 0x2205bdc0(570,801,600) bytes.
Signature 7f4696eb7960 (Vim::Event::HostEventArgument) has 13285493 instances taking 0x21b98d98(565,808,536) bytes.
Signature 7f4696eb78c0 (Vim::Event::ComputeResourceEventArgument) has 13280205 instances taking 0x21ddefb8(568,192,952) bytes.
Signature 562dc2a831f0 (Vmomi::DataArray<Vmomi::KeyAnyValue>) has 13086149 instances taking 0x21532ee8(559,099,624) bytes.
Signature 7f4696eb7d20 (Vim::Event::EventEx) has 13084710 instances taking 0xc31d4ee0(3,273,477,856) bytes.
Signature 7f4696eb7aa0 (Vim::Event::AlarmEventArgument) has 12934883 instances taking 0x20af9168(548,376,936) bytes.
Signature 562dc2ae8730 (Vmomi::Primitive<int>) has 12863119 instances taking 0x12707908(309,360,904) bytes.
Signature 7f4696eb4080 (Vim::Event::AlarmActionTriggeredEvent) has 4301050 instances taking 0x2b89b560(730,445,152) bytes.
Signature 7f4696eb4170 (Vim::Event::AlarmSnmpCompletedEvent) has 4301049 instances taking 0x272dbf98(657,309,592) bytes.

journalctl -xe

Jul 14 17:40:06 vCenter1 vpxd[20178]: Event [-22821230] [1-1] [2020-07-14T17:40:06.285759Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [SantaClara] [-22821230] [Alarm ‘Host hardware sensor state’ on host triggered an action]

Jul 14 17:40:06 vCenter1 vpxd[20178]: Event [-22821225] [1-1] [2020-07-14T17:40:06.286465Z] [vim.event.EventEx] [info] [] [SantaClara] [-22821225] [Alarm ‘Host hardware sensor state’ on host triggered by event -42004159 ‘Sensor -1 type , Description Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent #15 state assert for . Part Name/Number N/A N/A Manufacturer N/A’]

For 6 hours, the number of events is high

grep “Sensor -1 type” journalctl_-b–* | wc -l
371899

This is contributing towards, the VCDB growth.

VCDB=# SELECT nspname || ‘.’ || relname AS “relation”, pg_size_pretty(pg_total_relation_size(C.oid)) AS “total_size” FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (‘pg_catalog’, ‘information_schema’) AND C.relkind <> ‘i’ AND nspname !~ ‘^pg_toast’ ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;

The number of events in DB

VCDB=# SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;

[Action Plan]
https://kb.vmware.com/s/article/74607

As per VMware it is known issue with 6.7 causes the burst of events causing the high IO on the vCenter.
You could update all the ESXi to 6.7 P01 or the latest version to fix the issue or follow the workaround mentioned in the KB.

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VCSA6.7, VMware, VPXD | Tagged Vcenter Appliance Issue, vcenter issue, VMware logs, VPXD | Leave a comment

Steps to be taken while migrating the EC2 to different type.

Posted on June 30, 2020 by Ganadmin

We have migrated one of our EC2 instance from T2 to T3 and found there is a lack of performance on the server and when we had a call with AWS support they shared that the T3 type is based on the nitro hardware and it required drivers to support a nitro based hardware.

Downloaded and updated the network (ENA) and Storage (NVMe) along with the PV drivers on that instance.

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/migrating-latest-types.html#upgrade-pv

Posted in AWS, EC2 | Tagged AWS, AWS Server Migration Service | Leave a comment

Steps to recover the EC2 local windows administrator password

Posted on May 30, 2020 by Ganadmin

I took the image from the parent instance in which the key is working by using the .pem file but the instance which was created from the image the local administrator password is not working and it is failing with the below error.

After the investigation, I found that whenever we launch a new Windows instance using an Amazon-provided AMI, EC2Launch service is configured to generate a random password from the console.

However, after we launch the instance, this setting will be disabled on EC2Launch and you will need to enable it before creating a custom AMI. If this setting is not enabled on EC2Launch before creating the AMI, you won’t be able to retrieve the Password from the console and the same password of the source/parent instance would need to be used to access the new instance. Ideally, we should be able to login to this instance using the local Administrator password which was captured from the parent instance during the creation of the AMI.

As we are not able to login using the local administrator password of the parent AMI, we can use EC2-rescue tool to set random password and retrieve random generated password from the EC2 console using Key-pair for this instance.

Below Steps were shared from the Amazon support:

=========

Please follow steps to use EC2 Rescue to set password:

To troubleshoot this issue we used EC2Rescue tool and followed below steps:

[1] Launch helper instance from a Windows AWS Public AMI in the same VPC and subnet in which your current instance is launched.

[2] Detach the root volume of the instance and attach it to the helper instance as the secondary volume.

[3] Now, login to your helper Instance via RDP.

[4] Please download EC2Rescue tool by using the below link on the helper instance.

[+] https://urldefense.proofpoint.com/v2/url?u=https-3A__s3.amazonaws.com_ec2rescue_windows_EC2Rescue-5Flatest.zip&d=DwIFaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=ASemgKwYoytcsNGsg8cXc5YaWLC-52xuua2vDpCIG7A&m=m867DpcKhdYKq72sdzlFex1c08omm9ZHhWYu9MP8Mf4&s=8N6DFpivJTrWpZgHQFg6qGzMeFnjkNnjLf0KQLLTYII&e=

[5] Open EC2Rescue application -> Select the offline drive -> Click Diagnose and Rescue -> Select Ec2SetPassword

[6] Next, then Rescue, and OK for the volume to be offline.

[7] Once EC2Rescue has completed, detach the volume from the helper instance and re-attach the volume back to the original instance as /dev/sda1.

[8] Start the instance and Next, retrieve the password from the EC2 Console using your key-pair and connect to the instance.

Below links which have video included for the same steps.

[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__aws.amazon.com_premiumsupport_knowledge-2Dcenter_ec2rescue-2Dwindows-2Dtroubleshoot_&d=DwIFaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=ASemgKwYoytcsNGsg8cXc5YaWLC-52xuua2vDpCIG7A&m=m867DpcKhdYKq72sdzlFex1c08omm9ZHhWYu9MP8Mf4&s=3iaHYXJL_0UnkyB-H-31Ij5-kVOyQwtxpjiNAh1kj1U&e=

[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_WindowsGuide_ResettingAdminPassword-5FEC2Launch.html-23resetting-2Dpassword-2Dec2launch-2Dstep3&d=DwIFaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=ASemgKwYoytcsNGsg8cXc5YaWLC-52xuua2vDpCIG7A&m=m867DpcKhdYKq72sdzlFex1c08omm9ZHhWYu9MP8Mf4&s=WCfu0_HDpc-2Stn6AnsJPK2EMcBQnq9rUSZ9RI8e7yM&e=

Future reference:

=============

Going ahead, I suggest that you SysPrep an instance before creating its AMI so that password access is enabled and you are able to retrieve console generated password.

[+] Using SysPrep with EC2Launch: https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_WindowsGuide_ec2launch.html-23ec2launch-2Dsysprep&d=DwIFaQ&c=76Q6Tcqc-t2x0ciWn7KFdCiqt6IQ7a_IF9uzNzd_2pA&r=ASemgKwYoytcsNGsg8cXc5YaWLC-52xuua2vDpCIG7A&m=m867DpcKhdYKq72sdzlFex1c08omm9ZHhWYu9MP8Mf4&s=4py_NId0on0jc-DvpjVwp-0SJShdbKtNU78RiJ1lpdQ&e=