VCSA6.5 failed to login using AD credentials

Posted on November 30, 2019 by

One of our vCenter was having issue to login using the AD Credentials . We verified the DNS and the other VC ‘s which connects to the same DNS and AD , found no issues.

When we checked the websso.log , noticed the below error.

2019-11-25T16:08:43.717Z vsphere.local        8d2b3655-340a-46db-b879-5b680911c743 ERROR] [IdentityManager] Failed to authenticate principal [ADUSER@ADDOMAIN] for tenant [vsphere.local]com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 851968][null][null]

atcom.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(LinuxIdmNativeAdapter.java:180)
atcom.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.authenticate(ActiveDirectoryProvider.java:279)
atcom.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:2777)
atcom.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:9145)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$2.run(Unknown Source)
at sun.rmi.transport.Transport$2.run(Unknown Source)

We tried by rebooting the VC and also removing and adding the AD , even-though we are able to search the AD objects but the authentication was getting failed and finally  the below steps  fixed the issue.

  1. Removed the VC from the domain.
  2.  Deleted the computer account from the AD
  3.  Re-added the VC back to the domain.
  4.  Rebooted the VC, tested connection which was working fine.
Posted in Joining PSC with AD, Platform Services Controller (PSC ), Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , | Leave a comment

Error while migrating vSphere VMs to Amazon with AWS Server Migration Service

Posted on September 30, 2019 by

By the help of the link , configured the AWS Server Migration Service and at final stage of the sync it got failed with the error ” Instance failed to boot and establish network connectivity”

So we stopped all the non-microsoft services on the windows instance and tried the sync and it got completed successfully.

Posted in AWS, AWS Server Migration Service, VMware, Windows | Tagged , , | Leave a comment

Useful links to refer the CloudFormation functions.

Posted on August 31, 2019 by

AWS CloudFormation Templates – Sample CloudFormation templates, solutions, and snippets

CloudFormation::Init Documentation

Using AWS CloudFormer – Introduction and walkthrough for AWS CloudFormer

List of Services Supported in CloudFormation

JSON Formatter and Validator – Free tool for validating JSON templates

How to Use Nested Stacks – How to modularize your CloudFormation stacks

Conditionally-launch-aws-cloudformation-resources-based-on-user-input

Posted in AWS | Tagged , | Leave a comment

Solution for the SMB1 AD Authentication issue – ESXi 6.5U3 update

Posted on July 31, 2019 by

As per my previous blog on SMB1 AD authentication issue in 6.5u1 , VMware communicated that it will be fixed  after the 6.7 U2 update but it looks like in the recent 6.5 U3 update it got fixed.In the release notes they mentioned some fix related to AD and we tested with few hosts and able to connect the AD now without issue.

https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-65u3-release-notes.html

PR 2268193: Managing the Active Directory lwsmd service from the Host Client or vSphere Web Client might fail

Managing the Active Directory lwsmd service from the Host Client or vSphere Web Client might fail with the following error: Failed – A general system error occurred: Command /etc/init.d/lwsmd timed out after 30 secs.

This issue is resolved in this release.

Before:

 VMware ESXi 6.5.0 build-8294253

VMware ESXi 6.5.0 Update 2

After:

VMware ESXi 6.5.0 build-13932383

VMware ESXi 6.5.0 Update 3

Posted in ESX command, ESXi issue, ESXi Patches, VC6.0 Appliance Installation Issue, vCSA 6.0, VCSA6.5, VMware | Tagged , | Leave a comment

VCSA Shell is disabled.

Posted on June 30, 2019 by

One of our VC is down because of the space issue and not able to login to the Shell.

Followed the below steps to fix the same.

  • Accessed the VCSA on Grub mode
  • Found “/” and /storage/log partitions were full.
  • Released some space in both partitions and rebooted the VCSA
  • We were able to SSH into the VCSA , however found vmware-cm service was failing to start with below error:

Stderr = su: cannot not open session: Permission denied 

  • Checked root expiration date and found it was expired.
  • Reset root password.
  • Restarted services, but issue continue.
  • Tried rebooting the VCSA
  • All the services came up.
Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , , | Leave a comment

Useful links on various AWS topics

Posted on May 31, 2019 by

Centralized Logging – AWS Answers | https://aws.amazon.com/answers/logging/centralized-logging/

AWS Developer Forums: Discussion Forums | https://forums.aws.amazon.com/index.jspa

Amazon Web Services – Labs · GitHub | https://github.com/awslabs

GitHub – awslabs/aws-shell: An integrated shell for working with the AWS CLI. | https://github.com/awslabs/aws-shell

Region Table | https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/

AWS Regions and Endpoints – Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/rande.html

AWS Service Limits – Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html

AWS IP Address Ranges – Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

Error Retries and Exponential Backoff in AWS – Amazon Web Services |https://docs.aws.amazon.com/general/latest/gr/api-retries.html

Cloud Solutions by Application – Amazon Web Services (AWS) | https://aws.amazon.com/solutions/

AWS – Application Architecture Center | https://aws.amazon.com/architecture/

AWS Simple Icons | https://aws.amazon.com/architecture/icons/

Compliance Programs – Amazon Web Services (AWS) | https://aws.amazon.com/compliance/programs/

Case Studies & Customer Success – Amazon Web Services (AWS) | https://aws.amazon.com/solutions/case-studies

AWS Certification – AWS Cloud Computing Certification Program | https://aws.amazon.com/certification/

CI/CD on AWS

Click to access serverless-cicd-for-the-enterprise-on-the-aws-cloud.pdf

EC2 Template-  https://templates.cloudonaut.io/en/stable/ec2/

Posted in AWS | Tagged , | Leave a comment

EVC Mode on cluster for encryption

Posted on April 30, 2019 by

In our POC for our new application , noticed the significant differences on the performance on two different environment which is running on same hardware model.Even though the network layer is different from the environment A and B  the issue is when the data is copied locally .

Later we noticed when we change the EVC Mode from Intel Nehalem Generation to Intel Ivy Bridge a good performance improvement.

From the VMware blog  , it is mentioned that ” For most enterprise applications you can see there is no, or an almost immeasurable, performance impact when using EVC. But, there are certain corner cases, like encryption, that are crippled when instructions sets like AES-NI set are not available (Example: Oracle Transparent Data Encryption, OpenSSL)” so our data in POC is also encrypted and to make sure we setup the test VM and configured the Apache for the speed test with SSL on port 1000.

The htdocs folder was in localdisk (tintri NFS datastore) and tested with localdisk (local SAS disk datastore)

dlspeed=$(echo -n “scale=0; ” && curl -k https://localserver:1000/100MBfile -w “%{speed_download}” -o /dev/null -s | sed “s/\,/\./g” && echo “/1048576”); echo “$dlspeed” | bc -q

EVC mode disabled – 136MB/s 

EVC mode with “Intel Ivy Bridge” – 183MB/s

EVC mode with “Intel Nehalem Generation” – 41MB/s

As per the VMware Blog also they mentioned the test result and for encryption there is a huge different so when setting up the new cluster , we should understand the type of application and traffic type , based on that we have t select the cluster EVC mode.

Note : Anything above Nehalem Generation wont be supported by VMware 7.0 version for HP Gen8 old hardware models so pls check the VMware compatible guide and choose the EVC mode.

Refer:

Click to access vmware-vsphere-evc-performance-white-paper.pdf

https://blogs.vmware.com/vsphere/2014/06/enhanced-vmotion-compatibility-evc-affect-performance.html

Intel CPU EVC Matrix (VMware Enhanced vMotion Compatibility)

Posted in Dell, ESXi issue, HP, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , | Leave a comment

SRM IP customization issue on Redhat VMs

Posted on March 31, 2019 by

During the DR test we have noticed that few of  the Redhat  5\6\7 VMs were failing with various errors like ( “A general system error occurred: vix error codes = (3016,0)”,”Error – Timed out waiting for VMware Tools after 900 seconds”) on IP customization part and we have tried few option but it looks like VMware tools 10.1.x is the cause for the login delay and VMware has recommended to upgrade the SRM to 8.1.1 version.

IP customization might be failing due to the login delays with 10.1.x and above tools which uses SAML token authentication on the OS whereas 10.0.9 and below uses VIX authentication using VM tools when they are running on OS.

With SRM 8.1.1 we have added a delay in the code so the timeout value for this specific task is increased which can allow the customization script to be completed at OS level:

https://docs.vmware.com/en/Site-Recovery-Manager/8.1/rn/srm-releasenotes-8-1-1.html

{SRM 8.1.1 Release Notes – Refer “IP customization during recovery might fail due to delay in the completion of the SRM guest enrollment script”}

As a workaround, we can downgrade to VM tools 10.0.9 version until the upgrade can be performed.

Posted in SRM, VMware | Tagged , , , | Leave a comment

Bug in VCSA 6.5 U1\U2 which failed with invalid credentials on AD authentication

Posted on February 28, 2019 by

On of our vCenter was having issue on connecting the AD users and when users trying to connect the VC , it will fail with the invalid credentials error.

I have already mentioned few blogs about AD authentication  issue here and here .

Tried removing the AD and re-adding it from the PSC and also from the identify sources but it didn’t help to fix the issue so we started looking the logs and found the below error while trying to login using AD credentials.

vmware-sts-idmd.log:

2019-01-11T19:47:29.955Z vsphere.local        574439e1-8709-44ee-b5e8-a7ae7f0f8e14 ERROR] [ServerUtils] Exception ‘com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]’ com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]

As per the VMware below is the recommendations from them ..

  • “This is a known issue which has already been reported in VMware vCenter Server 6.5 Update 1. The workaround for this issue is for now is in, VMware vCenter Server 6.5 Update 1 Release Notes .
  • We still have the issue in VMware vCenter Server 6.5 Update 2.
  • Our engineering team is working on it.Once there is an update in future releases it will be updated “

Followed the steps below to workaround the issue.

Server Configuration Issues

  • In disjoint domain namespace the domain users might fail to authenticate after you update to vSphere 6.5 Update 1After you update a Platform Services Controller Appliance to vSphere 6.5 Update 1, in the disjoint domain namespace the users might fail  to authenticate.1. Log in to the Platform Services Controller Appliance as root and activate the bash shell.
    2. Leave the domain by running the /opt/likewise/bin/domainjoin-cli leave command.
    3. Reboot the appliance.
    4. Delete the computer account on the Active Directory.
    5. Log in to the appliance again and enable the bash shell.
    6. Join to the domain by running the following command /opt/likewise/bin/domainjoin-cli join domain-name domain_admin_user
    for example: /opt/likewise/bin/domainjoin-cli join vmware.com administrator
    7. Reboot the appliance.

Refer : VMware vCenter Server 6.5 Update 1 Release Notes ( Please check in release notes under Server Configuration Issues section)

https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-651-release-notes.html#server-configuration-issues-known

Posted in Joining PSC with AD, Platform Services Controller (PSC ), SSO, Vcenter Appliance, VCSA6.5, VMware | Tagged , , , , | Leave a comment

6.5u1 SMB1 issue with causes the AD authentication issue.

Posted on January 29, 2019 by

We had the AD authentication issue from the ESXi 6.5 U1 and tried various method mentioned in my previous blog but it got failed with all the options.

Below is the error while trying to connect the host from the domainjoin-cl cmd.

Error: LW_ERROR_KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN

Finally VMware engineering team has confirmed that the issue is because  of certain limitations in software that are affecting in the process of joining hosts to Domain.


Basically, smb1 must be enabled in DC in order to connect ESXi hosts to domain.
============
According to release notes for 6.5U1, SMB2 is supported.
https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-651-release-notes.html

Yes, SMB2 is supported from 6.5u1 onward but the initial SMB packet negotiation
request always happen over SMB1 packet. If SMB2 is enabled on both AD and the
host, then the negotiation switches to SMB2 otherwise it negotiates through SMB
packets only.
So if SMB1 is disabled on the domain controller then it would prevent the
initial packet negotiation, thus causing SMB packet drops and eventually domain
join failure with error ERROR_GEN_FAILURE.

From 6.7u2, we'll be supporting initial packet negotiation with SMB2 by default
instead of SMB1, thus disabling SMB1 completely.
============

We have also tried the option by selecting the preferred AD server option which is already enabled with  SMB1 , still we were not able to join in domain and got the update from the VMware as below..

"preferred server" option does not specifically imply that the connection will go through the server specified, but it's just a reference in case it is under the servers reported.
It seems like our only option would be enable SMB1 on the AD servers,

So, basically we cant be able to join these hosts to AD domain unless SMBv1 is enabled. Otherwise, we need to Wait for 6.7 U2 release.

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , , , , , | 1 Comment