I took the image from the parent instance in which the key is working by using the .pem file but the instance which was created from the image the local administrator password is not working and it is failing with the below error.
After the investigation, I found that whenever we launch a new Windows instance using an Amazon-provided AMI, EC2Launch service is configured to generate a random password from the console.
However, after we launch the instance, this setting will be disabled on EC2Launch and you will need to enable it before creating a custom AMI. If this setting is not enabled on EC2Launch before creating the AMI, you won’t be able to retrieve the Password from the console and the same password of the source/parent instance would need to be used to access the new instance. Ideally, we should be able to login to this instance using the local Administrator password which was captured from the parent instance during the creation of the AMI.
As we are not able to login using the local administrator password of the parent AMI, we can use EC2-rescue tool to set random password and retrieve random generated password from the EC2 console using Key-pair for this instance.
Below Steps were shared from the Amazon support:
=========
Please follow steps to use EC2 Rescue to set password:
To troubleshoot this issue we used EC2Rescue tool and followed below steps:
[1] Launch helper instance from a Windows AWS Public AMI in the same VPC and subnet in which your current instance is launched.
[2] Detach the root volume of the instance and attach it to the helper instance as the secondary volume.
[3] Now, login to your helper Instance via RDP.
[4] Please download EC2Rescue tool by using the below link on the helper instance.
Going ahead, I suggest that you SysPrep an instance before creating its AMI so that password access is enabled and you are able to retrieve console generated password.
Amazon FSx for Windows File Server provides a fully managed native Microsoft Windows file system. Amazon FSx provides NTFS file systems that can be accessed from up to thousands of compute instances using the SMB protocol (SMB 2.0 to 3.1.1). You can access your Amazon FSx file system from Amazon EC2, VMware Cloud on AWS, Amazon WorkSpaces, and Amazon AppStream 2.0 instances.
The service works with Microsoft Active Directory (AD) to integrate your file system with your existing Windows environments. Amazon FSx uses Distributed File System (DFS) Replication to support multi-AZ deployments. To ensure compatibility with your applications, Amazon FSx supports all Windows versions starting from Windows Server 2008 and Windows 7, and also current versions of Linux.
AWS FSx can be used for various application workloads like home directories, web serving & content management, enterprise applications, analytics, and media & entertainment. All Amazon FSx file system data is automatically encrypted at rest and in transit.
1.0 FSx for Windows file system
Prerequisites – Currently AWS FSx only work with AWS Managed AD [AWS MAD], AWS support AWS Directory Service’s AD sharing feature, AWS plan to support AD Connector and Self-managed Microsoft Active Directory.
These are the Directory Types options AWS have –
a) AWS Managed Microsoft AD
b) Simple AD
c) AD Connector
d) Cloud Directory
e) Amazon Cognito Your User Pools
No need to select any option from above.Self-managed Microsoft Active Directory is something we need to use for our use cases. Other options are little complicated and has some overhead to it to maintain as explained below.
a) Create AWS Managed Active Directory
We need to create AD (eg: xyz.storage.com). During AWS Managed Directory service creation, you will be asked for – Directory Type; Edition; Directory DNS name; Directory NetBIOS name; VPC; Subnet; AZ.
Limitation:
Here we need to maintain AD in respective account. So, this AD need a continues sync for latest AD objects.
Details explained here for the filesystem creation for this method
b) Simple AD
Simple AD is a standalone managed directory that is powered by a Linux-Samba Active Directory–compatible server. Not recommended as it needs to create a directory and object limitation
b) AD Connector
Need to create a trust relationship between source (app account ) and destination account (AD account).
–> Need to create AWS Managed AD directory ID
Limitation:
1) Overhead for this process and every account using this will need to have this trust relation.
2) Cost for active directory service created.
d) Cloud Directory : Similar to RDS cloud directory, Cloud Directory is a high-performance, serverless, hierarchical data store still same limitations as above mentioned for Simple AD.
e) Amazon Cognito Your User Pools : Directs to Cognito service for directory creation. No use
Share folder will comes by default and cannot be deleted.
2.0 Automatic Daily Backups
Amazon FSx automatically takes backups of your file systems once a day. These daily backups are taken during the daily backup window that was established when you created the file system. At some point during the daily backup window, storage I/O might be suspended briefly while the backup process initializes (typically under a few seconds). When you choose your daily backup window, we recommend that you choose a convenient time of the day outside of the normal operating hours for the applications that will use the file system. Backups are kept for a certain period of time, known as a retention period. By default, backups are retained for 7 days. However, you can change the retention period to anywhere in a range of 0–35 days.
We can perform backup creation and restoration from FSx Management Console, the AWS CLI, or one of the AWS SDKs
3.0 Multi-AZ File System Deployments
For workloads that require multi-AZ redundancy to tolerate temporary AZ unavailability, We can create multiple file systems in separate AZs, keep them in sync, and configure failover between them. Amazon FSx fully supports the use of the Microsoft Distributed File System (DFS) for file system deployments across multiple AZs to get Multi-AZ availability and durability. Using DFS Replication, you can automatically replicate data between two file systems. Using DFS Namespaces, you can configure one file system as your primary and the other as your standby, with automatic failover to the standby in the event that the primary becomes unresponsive. MS DFS support both async and sync replication.
AWS FSx provides high availability and failover support across multiple AZs which can be used for shared storage and also as mapped drive instead of EBS volumes as EBS cannot span Multi-AZ.
4.0 Benefits and Cons of FSx
Benefits:
· AWS FSx is fully managed. It relies on SSD storage and performs with high levels of IOPS and throughput, as well as consistent sub-millisecond latencies for a well-designed infra.
· AWS FSx is secure. All of the file systems are a part of the Virtual Private Cloud (VPC); all data is encrypted both in transit and at rest, and all activities are logged to CloudTrail
Cons:
· AWS FSx for windows File server supports custo DNS only Single-AZ filesystems, not for Multi-AZ as if yet
Recently we have upgraded the VMtools version from 10.305\10.346 to 11265 – 11.0.1 and we noticed few VMs went to hung status and noticed the below alert in windows VMs.
vmware.log:
2020-02-07T12:50:58.182Z| vcpu-0| I125: Guest: vsep: AUDIT: VFileSocketMgrCloseSocket : Mux is disconnected <—————————————— 2020-02-07T12:50:58.297Z| vmx| I125: VigorTransportProcessClientPayload: opID=3997b233-39-9b26 seq=290: Receiving MKS.IssueTicket request. 2020-02-07T12:50:58.297Z| vmx| I125: SOCKET 5 (129) creating new listening socket on port -1 2020-02-07T12:50:58.297Z| vmx| I125: Issuing new webmks ticket a9161e… (120 seconds) 2020-02-07T12:50:58.297Z| vmx| I125: VigorTransport_ServerSendResponse opID=3997b233-39-9b26 seq=290: Completed MKS request. 2020-02-07T12:50:58.666Z| vcpu-0| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0 2020-02-07T12:50:58.667Z| vcpu-1| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0 2020-02-07T12:50:58.676Z| vcpu-1| I125: Guest: vsep: AUDIT: SetupConsumerContext : Setting event Type as 256 from 0
VMware ticket has been raised and they recommended to upgrade the NSX Manager to 6.4.4 and confirmed the below
There is an internal bug which confirms that this is a known issue with VMware tools version you are using ( 11.0.1 ) and there is no external documentation available confirming this aspect. We have confirmed based on an engineering ticket that we have referred. As per the engineering ticket, this should be made available in the release notes of 11.0.5 and expected to be fixed in 11.1. There is no ETA mentioned about these releases.
2019 started with the lot of new surprises in the company roadmap and one of the main change is to move the on-perm to the cloud which means reducing the VMware footprint. Initially it was tough to accept the change but it was good planing on the change from the management end like providing enough training on the AWS\Azur and once team is having sufficient knowledge and confident then start on migrating the work load to the cloud.
I was put in AWS Training and lot of new learnings which allowed to prepare for the certification and after several months of preparation and experience, completed the AWS solution architect certification. Last three months started migrating the application to AWS which is very challenging to understand the current design of the application and planning on it to run in cloud.Gained some experience in CHEF and last two months started working on CI\CD also with few python automation.
Past four years I got the opportunity to attend the VMware conference but this year since the focus is on the cloud, didnt get the chance to attend but at the same time had change to attend my first AWS-Reinvent which is awesome for learning and explore new services in AWS..
It was a good year and looking forward 2020 to learn more in cloud services.
One of our vCenter was having issue to login using the AD Credentials . We verified the DNS and the other VC ‘s which connects to the same DNS and AD , found no issues.
When we checked the websso.log , noticed the below error.
2019-11-25T16:08:43.717Z vsphere.local 8d2b3655-340a-46db-b879-5b680911c743 ERROR] [IdentityManager] Failed to authenticate principal [ADUSER@ADDOMAIN] for tenant [vsphere.local]com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 851968][null][null]
atcom.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(LinuxIdmNativeAdapter.java:180)
atcom.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.authenticate(ActiveDirectoryProvider.java:279)
atcom.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:2777)
atcom.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:9145)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$2.run(Unknown Source)
at sun.rmi.transport.Transport$2.run(Unknown Source)
We tried by rebooting the VC and also removing and adding the AD , even-though we are able to search the AD objects but the authentication was getting failed and finally the below steps fixed the issue.
Removed the VC from the domain.
Deleted the computer account from the AD
Re-added the VC back to the domain.
Rebooted the VC, tested connection which was working fine.
By the help of the link , configured the AWS Server Migration Service and at final stage of the sync it got failed with the error ” Instance failed to boot and establish network connectivity”
So we stopped all the non-microsoft services on the windows instance and tried the sync and it got completed successfully.
As per my previous blog on SMB1 AD authentication issue in 6.5u1 , VMware communicated that it will be fixed after the 6.7 U2 update but it looks like in the recent 6.5 U3 update it got fixed.In the release notes they mentioned some fix related to AD and we tested with few hosts and able to connect the AD now without issue.
PR 2268193: Managing the Active Directory lwsmd service from the Host Client or vSphere Web Client might fail
Managing the Active Directory lwsmd service from the Host Client or vSphere Web Client might fail with the following error: Failed – A general system error occurred: Command /etc/init.d/lwsmd timed out after 30 secs.
