AWS Compute related updates

AWS End of Support Migration Program for Windows Server now available as a self-serve solution for customers

Resource Access Manager Support is now available on AWS Outposts

New course for Amazon Elastic Kubernetes Service

Amazon EKS now supports Kubernetes version 1.18

AWS Lambda Extensions: a new way to integrate Lambda with operational tools

AWS Compute Optimizer enhances EC2 instance type recommendations with Amazon EBS metrics

Amazon EBS CSI driver now supports AWS Outposts

Amazon ElastiCache on Outposts is now available

AWS Elastic Beanstalk Adds Support for Running Multi-Container Applications on AL2 based Docker Platform

AWS Batch introduces tag-based access control

Amazon EC2 G4dn Bare Metal Instances with NVIDIA T4 Tensor Core GPUs, now available in 15 additional regions

AWS Launch Wizard now supports SAP HANA backups with AWS Backint Agent

Posted in AWS, EC2 | Tagged | Leave a comment

Steps to blacklist the problematic DCs in VMware VCSA 6.7U3

We had a DNS issue in one of the DC running active directory integrated  DNS service and it caused our vCenter to fail to connect the domain in AD so we have changed the  DNS to the IPs which is working properly but identified still AD authentication getting failed and in the VAR\LOG\Messages it was still pointing to the problematic DC and failing to authenticate.

After a few research got the instruction from the VCSA6.7 U3b release notes about the steps to blacklist the DCs and added the problematic DC IP as mentioned below.

Active Directory authentication or joining a domain is slow

Active Directory authentication or joining a domain might be slow when configured with Integrated Windows Authentication (IWA), because of infrastructure issues such as network latency and firewalls in some of the domain controllers.

This issue is resolved in this release. The fix provides the option to blacklist selected domain controllers in case of infrastructure issues.

To set the option, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs DC_IP1,DC_IP2,...
# /opt/likewise/bin/lwsm restart lwreg

To revert to the default settings, use the following commands:
# /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\netlogon\Parameters]' BlacklistedDCs ""
# /opt/likewise/bin/lwsm restart lwreg

But still we noticed the VC is connecting to the problematic DC and also in the file /var/lib/likewise/krb5-affinity.conf it was showing the problamatic DC IP and when we tried to change it manually , automatically it got updated to the old problamatic DC IP .

After research we added the VC subnet in Active Directory Sites and Services to the new DCs and waited for few mins and noticed in the krb5-affinity.conf the new DC IPs got updated and issue got fixed by pointing the VC to the correct DC and ignoring the problematic DC.

Note : BlacklistDCs will work only from the 6.7U3b version.

Useful links:

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VCSA6.7, VMware | Tagged , , , | Leave a comment

cfn-lint useful tool for the CloudFormation.

Recently had the chance to learn about the CFN-LINT tool which is a very useful tool to validate the CloudFormation template directly using the editor and it makes us create the CF without any error and secured.

Posted in AWS, AWS Server Migration Service, EC2 | Tagged , | Leave a comment

vCenter VPXD crashes because of high memory.

We have a vcenter environment with around  500 ESXi hosts running on multiple clusters and for the past several weeks we had the issue of Vcetner down because of VPXD crash and the service will be in stopped status.

VMware support identified the VCDB growth is huge with high CPU and memory usage on the vCenter and they started to investigate the same.

Most of the memory usage is contributed by the Events

Signature 7f46978acb30 (Vmomi::KeyAnyValue) has 81430551 instances taking 0xd0aaf2b8(3,500,864,184) bytes.
Signature 562dc2a74a90 (Vmomi::Primitive<std::string>) has 55823822 instances taking 0x50096420(1,342,792,736) bytes.
Signature 7f4696eb77d0 (Vim::Event::ManagedEntityEventArgument) has 21574134 instances taking 0x37913be0(932,264,928) bytes.
Signature 7f4696eb7870 (Vim::Event::DatacenterEventArgument) has 13307524 instances taking 0x2205bdc0(570,801,600) bytes.
Signature 7f4696eb7960 (Vim::Event::HostEventArgument) has 13285493 instances taking 0x21b98d98(565,808,536) bytes.
Signature 7f4696eb78c0 (Vim::Event::ComputeResourceEventArgument) has 13280205 instances taking 0x21ddefb8(568,192,952) bytes.
Signature 562dc2a831f0 (Vmomi::DataArray<Vmomi::KeyAnyValue>) has 13086149 instances taking 0x21532ee8(559,099,624) bytes.
Signature 7f4696eb7d20 (Vim::Event::EventEx) has 13084710 instances taking 0xc31d4ee0(3,273,477,856) bytes.
Signature 7f4696eb7aa0 (Vim::Event::AlarmEventArgument) has 12934883 instances taking 0x20af9168(548,376,936) bytes.
Signature 562dc2ae8730 (Vmomi::Primitive<int>) has 12863119 instances taking 0x12707908(309,360,904) bytes.
Signature 7f4696eb4080 (Vim::Event::AlarmActionTriggeredEvent) has 4301050 instances taking 0x2b89b560(730,445,152) bytes.
Signature 7f4696eb4170 (Vim::Event::AlarmSnmpCompletedEvent) has 4301049 instances taking 0x272dbf98(657,309,592) bytes.

journalctl -xe

Jul 14 17:40:06 vCenter1 vpxd[20178]: Event [-22821230] [1-1] [2020-07-14T17:40:06.285759Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [SantaClara] [-22821230] [Alarm ‘Host hardware sensor state’ on host  triggered an action]

Jul 14 17:40:06 vCenter1 vpxd[20178]: Event [-22821225] [1-1] [2020-07-14T17:40:06.286465Z] [vim.event.EventEx] [info] [] [SantaClara] [-22821225] [Alarm ‘Host hardware sensor state’ on host triggered by event -42004159 ‘Sensor -1 type , Description Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent #15 state assert for . Part Name/Number N/A N/A Manufacturer N/A’]

For 6 hours, the number of events is high

grep “Sensor -1 type” journalctl_-b–* | wc -l

This is contributing towards, the VCDB growth.

VCDB=# SELECT nspname || ‘.’ || relname AS “relation”, pg_size_pretty(pg_total_relation_size(C.oid)) AS “total_size” FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (‘pg_catalog’, ‘information_schema’) AND C.relkind <> ‘i’ AND nspname !~ ‘^pg_toast’ ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;

relation       | total_size
vc.vpx_event_arg_35 | 9661 MB
vc.vpx_event_arg_34 | 9568 MB
vc.vpx_event_arg_40 | 9543 MB
vc.vpx_event_arg_32 | 9427 MB
vc.vpx_event_arg_37 | 9082 MB
vc.vpx_event_arg_38 | 8742 MB
vc.vpx_event_arg_36 | 8721 MB
vc.vpx_event_arg_39 | 8249 MB
vc.vpx_event_arg_33 | 8169 MB
vc.vpx_event_arg_57 | 7957 MB

The number of events in DB


numevents |                 event_type                 |         username
58907485 | vim.event.AlarmActionTriggeredEvent        |
58905170 | |
58899874 | vim.event.AlarmSnmpCompletedEvent          |
5894366 |       |
5892674 |    |
5892554 |         |
1564588 | vim.event.AlarmStatusChangedEvent          |
702653 | vim.event.BadUsernameSessionEvent          | root
694115 | esx.audit.account.locked                   |
300159 | vim.event.TaskEvent                        |

[Action Plan]

As per VMware it is known issue with 6.7 causes the burst of events causing the high IO on the vCenter.
You could update all the ESXi to 6.7 P01 or the latest version to fix the issue or follow the workaround mentioned in the KB.

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VCSA6.7, VMware, VPXD | Tagged , , , | Leave a comment

Steps to be taken while migrating the EC2 to different type.

We have migrated one of our EC2 instance from T2 to T3 and found there is a lack of performance on the server and when we had a call with AWS support they shared that the T3 type is based on the nitro hardware and it required drivers to support a nitro based hardware.

Downloaded and updated the network (ENA) and Storage (NVMe) along with the PV drivers on that instance.

Posted in AWS, EC2 | Tagged , | Leave a comment

Some of the free stuffs which we can use it during this COVID-19 lockdown

Some of the free stuffs are available which is in the list below and hoping it will help during this tough period.


Google Cloud:


Books: Kindle Free














Kids Story :

Sling : Get 14 days of access with no credit card required


HBO : HBO unlocks 500 hours of must-see films and TV series to stream for free

Indian Movies : One month free

Posted in AWS, General | Tagged , , | Leave a comment

Amazon FSx for Windows File Server


Amazon FSx for Windows File Server provides a fully managed native Microsoft Windows file system. Amazon FSx provides NTFS file systems that can be accessed from up to thousands of compute instances using the SMB protocol (SMB 2.0 to 3.1.1).  You can access your Amazon FSx file system from Amazon EC2, VMware Cloud on AWS, Amazon WorkSpaces, and Amazon AppStream 2.0 instances. 

 The service works with Microsoft Active Directory (AD) to integrate your file system with your existing Windows environments. Amazon FSx uses Distributed File System (DFS) Replication to support multi-AZ deployments. To ensure compatibility with your applications, Amazon FSx supports all Windows versions starting from Windows Server 2008 and Windows 7, and also current versions of Linux. 

AWS FSx can be used for various application workloads like home directories, web serving & content management, enterprise applications, analytics, and media & entertainment. All Amazon FSx file system data is automatically encrypted at rest and in transit.  

1.0 FSx for Windows file system 

Prerequisites – Currently AWS FSx only work with AWS Managed AD [AWS MAD], AWS support AWS Directory Service’s AD sharing feature, AWS plan to support AD Connector and Self-managed Microsoft Active Directory.  

These are the Directory Types options AWS have –  

a)       AWS Managed Microsoft AD 

b)      Simple AD 

c)       AD Connector 

d)      Cloud Directory

e)      Amazon Cognito Your User Pools 

No need to select any option from above. Self-managed Microsoft Active Directory is something we need to use for our use cases. Other options are little complicated and has some overhead to it to maintain as explained below.

a)        Create AWS Managed Active Directory  

We need to  create AD (eg: During AWS Managed Directory service creation, you will be asked for – Directory Type; Edition; Directory DNS name; Directory NetBIOS name; VPC; Subnet; AZ. 


 Here we need to maintain AD in respective account. So, this AD need a continues sync for latest AD objects.

Details explained here for the filesystem creation for this method

b)      Simple AD 

Simple AD is a standalone managed directory that is powered by a Linux-Samba Active Directory–compatible server. Not recommended as it needs to create a directory and object limitation

b)      AD Connector 

Need to create a trust relationship between source (app account ) and destination account (AD account).

–> Need to create AWS Managed AD directory ID


1)      Overhead for this process and every account using this will need to have this trust relation.

2)      Cost for active directory service created.

d) Cloud Directory : Similar to RDS cloud directory, Cloud Directory is a high-performance, serverless, hierarchical data store still same limitations as above mentioned for Simple AD.

e) Amazon Cognito Your User Pools : Directs to Cognito service for directory creation. No use

Self-managed Microsoft Active Directory: this is recommended to use

1)      Select FSx for windows file server

Use cases for both 

  Amazon FSx for Lustre Amazon FSx for Windows File Server
Performance Compute Intensive Simple fileshare
Lifecycle Yes No lifecycle
Storage Minimum 1.2 TB Minimum 32 gig
AD authentecation NO Yes
Price $0.14 GB-month / 30 / 24 = $0.000194/GB-hour

3600 GB x $0.000194/GB-hour x 72 hours = $50.40

Total FSx for Lustre charge for 72 hours = $50.40
Storage: 1,024 GB-months x $0.130 GB-month= $133/mo

Throughput: 8 MBPS-months x $2.200/MBps-month= $18/mo

Backup: 500 GB-months x $0.050/GB-month = $25/mo

Total monthly charge: $176 ($0.172/GB-mo)

2)      Then for filesystem creation mention below options

File System Name – This will not be used to access the file share or File System. 

Storage Capacity – Minimum 32 GiB; Maximum 65,536 GiB 

Throughput capacity – Recommended is 8MB/s and max can go up to 2048MB/s .

3)      Network info

Please remember the fs is AZ specific but accessible in all Az’s of a region. Need to DFS for redundancy as explained below in doc.

Network & Security – Select your VPC; Availability zone; Subnet; VPC Security Group 

4)      Select the Window authentication method

Select Self-managed AD (SMAD), if selected AWS MAD-id  then create directory services mentioned above methods.

Mention all the requested info :- FDQN, Service account (use ’id” : “serviceaccount”,

Use  Route53 revolvers route the traffice to DNS

5)      Other options

OU :- Please don’t leave default blank or else all the hardening rules will be applied to this , not sure on what impact it might have

OU=Storage and Backup,OU=Appliance,OU=Computers,OU=Objects,DC=ads,DC=xyz,DC=com

Encryption – Default [AWS Key Management Service (KMS) encryption key that protects your file system data at rest] 

6)      Review summary and click “create filesystem”

7) Filesystem will be created

Check via using DNS name


Share folder will comes by default and cannot be deleted.

2.0 Automatic Daily Backups 

 Amazon FSx automatically takes backups of your file systems once a day. These daily backups are taken during the daily backup window that was established when you created the file system. At some point during the daily backup window, storage I/O might be suspended briefly while the backup process initializes (typically under a few seconds). When you choose your daily backup window, we recommend that you choose a convenient time of the day outside of the normal operating hours for the applications that will use the file system. Backups are kept for a certain period of time, known as a retention period. By default, backups are retained for 7 days. However, you can change the retention period to anywhere in a range of 0–35 days. 

We can perform backup creation and restoration from FSx Management Console, the AWS CLI, or one of the AWS SDKs 

3.0 Multi-AZ File System Deployments 

For workloads that require multi-AZ redundancy to tolerate temporary AZ unavailability, We can create multiple file systems in separate AZs, keep them in sync, and configure failover between them. Amazon FSx fully supports the use of the Microsoft Distributed File System (DFS) for file system deployments across multiple AZs to get Multi-AZ availability and durability. Using DFS Replication, you can automatically replicate data between two file systems. Using DFS Namespaces, you can configure one file system as your primary and the other as your standby, with automatic failover to the standby in the event that the primary becomes unresponsive. MS DFS support both async and sync replication. 

      AWS FSx provides high availability and failover support across multiple AZs which can be used for shared storage and also as mapped drive instead of EBS volumes as EBS cannot span Multi-AZ. 

4.0 Benefits and Cons of FSx


·      AWS FSx is fully managed. It relies on SSD storage and performs with high levels of IOPS and throughput, as well as consistent sub-millisecond latencies for a well-designed infra.

·      AWS FSx is secure. All of the file systems are a part of the Virtual Private Cloud (VPC); all data is encrypted both in transit and at rest, and all activities are logged to CloudTrail


·      AWS FSx for windows File server supports custo DNS only Single-AZ filesystems, not for Multi-AZ as if yet

·      Cost is not accurate enough currently

AWS Documentation

Posted in AWS, Windows | Tagged , , , | Leave a comment

Bug in network introspection driver (vnetflt.sys) VMtools 11265 (11.0.1)

Recently we have upgraded the VMtools version from 10.305\10.346 to 1126511.0.1 and we noticed few VMs went to hung status and noticed the below alert in windows VMs.


2020-02-07T12:50:58.182Z| vcpu-0| I125: Guest: vsep: AUDIT: VFileSocketMgrCloseSocket : Mux is disconnected <——————————————
2020-02-07T12:50:58.297Z| vmx| I125: VigorTransportProcessClientPayload: opID=3997b233-39-9b26 seq=290: Receiving MKS.IssueTicket request.
2020-02-07T12:50:58.297Z| vmx| I125: SOCKET 5 (129) creating new listening socket on port -1
2020-02-07T12:50:58.297Z| vmx| I125: Issuing new webmks ticket a9161e… (120 seconds)
2020-02-07T12:50:58.297Z| vmx| I125: VigorTransport_ServerSendResponse opID=3997b233-39-9b26 seq=290: Completed MKS request.
2020-02-07T12:50:58.666Z| vcpu-0| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0
2020-02-07T12:50:58.667Z| vcpu-1| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0
2020-02-07T12:50:58.676Z| vcpu-1| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0

VMware ticket has been raised and they recommended to upgrade the NSX Manager to 6.4.4 and confirmed the below

There is an internal bug which confirms that this is a known issue with VMware tools version you are using ( 11.0.1 ) and there is no external documentation available confirming this aspect. We have confirmed based on an engineering ticket that we have referred. As per the engineering ticket, this should be made available in the release notes of 11.0.5 and expected to be fixed in 11.1. There is no ETA mentioned about these releases.

Posted in ESXi issue, NSX 6.1.4, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMs, VMtools issue, VMware, Vnic, vShield, Windows | Tagged , , | Leave a comment

Useful links on EC2\FSX with new features

Posted in AWS, EC2 | Tagged , | Leave a comment