SRM IP customization issue on Redhat VMs

Posted on March 31, 2019 by

During the DR test we have noticed that few of  the Redhat  5\6\7 VMs were failing with various errors like ( “A general system error occurred: vix error codes = (3016,0)”,”Error – Timed out waiting for VMware Tools after 900 seconds”) on IP customization part and we have tried few option but it looks like VMware tools 10.1.x is the cause for the login delay and VMware has recommended to upgrade the SRM to 8.1.1 version.

IP customization might be failing due to the login delays with 10.1.x and above tools which uses SAML token authentication on the OS whereas 10.0.9 and below uses VIX authentication using VM tools when they are running on OS.

With SRM 8.1.1 we have added a delay in the code so the timeout value for this specific task is increased which can allow the customization script to be completed at OS level:

https://docs.vmware.com/en/Site-Recovery-Manager/8.1/rn/srm-releasenotes-8-1-1.html

{SRM 8.1.1 Release Notes – Refer “IP customization during recovery might fail due to delay in the completion of the SRM guest enrollment script”}

As a workaround, we can downgrade to VM tools 10.0.9 version until the upgrade can be performed.

Posted in SRM, VMware | Tagged , , , | Leave a comment

Bug in VCSA 6.5 U1\U2 which failed with invalid credentials on AD authentication

Posted on February 28, 2019 by

On of our vCenter was having issue on connecting the AD users and when users trying to connect the VC , it will fail with the invalid credentials error.

I have already mentioned few blogs about AD authentication  issue here and here .

Tried removing the AD and re-adding it from the PSC and also from the identify sources but it didn’t help to fix the issue so we started looking the logs and found the below error while trying to login using AD credentials.

vmware-sts-idmd.log:

2019-01-11T19:47:29.955Z vsphere.local        574439e1-8709-44ee-b5e8-a7ae7f0f8e14 ERROR] [ServerUtils] Exception ‘com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]’ com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]

As per the VMware below is the recommendations from them ..

  • “This is a known issue which has already been reported in VMware vCenter Server 6.5 Update 1. The workaround for this issue is for now is in, VMware vCenter Server 6.5 Update 1 Release Notes .
  • We still have the issue in VMware vCenter Server 6.5 Update 2.
  • Our engineering team is working on it.Once there is an update in future releases it will be updated “

Followed the steps below to workaround the issue.

Server Configuration Issues

  • In disjoint domain namespace the domain users might fail to authenticate after you update to vSphere 6.5 Update 1After you update a Platform Services Controller Appliance to vSphere 6.5 Update 1, in the disjoint domain namespace the users might fail  to authenticate.1. Log in to the Platform Services Controller Appliance as root and activate the bash shell.
    2. Leave the domain by running the /opt/likewise/bin/domainjoin-cli leave command.
    3. Reboot the appliance.
    4. Delete the computer account on the Active Directory.
    5. Log in to the appliance again and enable the bash shell.
    6. Join to the domain by running the following command /opt/likewise/bin/domainjoin-cli join domain-name domain_admin_user
    for example: /opt/likewise/bin/domainjoin-cli join vmware.com administrator
    7. Reboot the appliance.

Refer : VMware vCenter Server 6.5 Update 1 Release Notes ( Please check in release notes under Server Configuration Issues section)

https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-651-release-notes.html#server-configuration-issues-known

Posted in Joining PSC with AD, Platform Services Controller (PSC ), SSO, Vcenter Appliance, VCSA6.5, VMware | Tagged , , , , | Leave a comment

6.5u1 SMB1 issue with causes the AD authentication issue.

Posted on January 29, 2019 by

We had the AD authentication issue from the ESXi 6.5 U1 and tried various method mentioned in my previous blog but it got failed with all the options.

Below is the error while trying to connect the host from the domainjoin-cl cmd.

Error: LW_ERROR_KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN

Finally VMware engineering team has confirmed that the issue is because  of certain limitations in software that are affecting in the process of joining hosts to Domain.


Basically, smb1 must be enabled in DC in order to connect ESXi hosts to domain.
============
According to release notes for 6.5U1, SMB2 is supported.
https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-651-release-notes.html

Yes, SMB2 is supported from 6.5u1 onward but the initial SMB packet negotiation
request always happen over SMB1 packet. If SMB2 is enabled on both AD and the
host, then the negotiation switches to SMB2 otherwise it negotiates through SMB
packets only.
So if SMB1 is disabled on the domain controller then it would prevent the
initial packet negotiation, thus causing SMB packet drops and eventually domain
join failure with error ERROR_GEN_FAILURE.

From 6.7u2, we'll be supporting initial packet negotiation with SMB2 by default
instead of SMB1, thus disabling SMB1 completely.
============

We have also tried the option by selecting the preferred AD server option which is already enabled with  SMB1 , still we were not able to join in domain and got the update from the VMware as below..

"preferred server" option does not specifically imply that the connection will go through the server specified, but it's just a reference in case it is under the servers reported.
It seems like our only option would be enable SMB1 on the AD servers,

So, basically we cant be able to join these hosts to AD domain unless SMBv1 is enabled. Otherwise, we need to Wait for 6.7 U2 release.

Posted in ESXi issue, Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , , , , , | 1 Comment

2018 – Blogs which helped me to gain knowledge.

Posted on December 31, 2018 by

Below is the list of the blogs which helped me to gain knowledge on some new technologies in 2018.

https://theithollow.com/2017/06/26/migrate-vsphere-vms-amazon-aws-server-migration-service/

http://mistwire.com/2017/09/use-aws-lambda-to-schedule-instance-startsstops-and-save-some-money/

https://theithollow.com/2017/03/06/using-packer-create-vsphere-aws-images/

https://theithollow.com/2017/12/11/use-amazon-cloudwatch-logs-metric-filters-send-alerts/

https://jsimon-public.s3.amazonaws.com/reinvent2017.html

https://www.portal.reinvent.awsevents.com/connect/search.ww#loadSearch-searchPhrase=vmware&searchType=session&tc=0&sortBy=abbreviationSort&p=%3Fsc_channel

https://cloud.vmware.com/community/2017/12/04/using-aws-vpc-endpoint-access-data-s3-spark-vmware-cloud-aws-2/

https://acloud.guru/courses

https://aws.amazon.com/blogs/developer/announcing-lambda-support-for-powershell-core/?sc_channel=sm&sc_campaign=Serverless,Developer_Blog&sc_publisher=TWITTER&sc_country=SDK&sc_geo=GLOBAL&sc_outcome=awareness&trk=_TWITTER&sc_content=PowerShell&sc_category=AWS_Lambda&linkId=56724189

https://sqldbawithabeard.com/2018/12/24/running-windows-and-linux-sql-containers-together/

https://code.vmware.com/samples?id=5121

https://blogs.vmware.com/vsphere/2018/12/the-vsphere-upgrade-blog-series-wrap-up.html

https://docs.microsoft.com/en-au/azure/architecture/aws-professional/?wt.mc_id=MVP

https://blogs.vmware.com/vsphere/2018/10/vcenter-server-windows-migrations.html

https://graham-beer.github.io/2018/AWS-Lambda-with-PowerShell-1/

https://docs.microsoft.com/en-us/teamblog/introducing-ms-learn?wt.mc_id=docsmsft-twitter

https://vports.info/ports/

https://vmiss.net/2018/09/19/how-to-get-started-with-vmware-vsphere-security/

https://docs.microsoft.com/en-us/azure/site-recovery/migrate-tutorial-windows-server-2008

https://blogs.vmware.com/vsphere/2018/09/introduction-to-automating-your-vsphere-upgrade.html?src=so_5a314d0604f0c&cid=70134000001SluU

https://domalab.com/copy-files-veeam-console/?platform=hootsuite

https://www.thomasmaurer.ch/2018/09/windows-server-2019-system-insights/

http://www.lucd.info/2018/08/05/invoke-vmscriptplus-v2/

https://github.com/tpcarman/As-Built-Report

https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker?view=sql-server-2017

http://www.vmspot.com/monitoring-vcsa-filesystem-with-vrealize-operations/

https://blogs.vmware.com/code/2018/07/24/automate-file-based-backup-vcsa-rest/

https://jwmoss.github.io/blog/automating-template-builds-with-packer.html

https://jwmoss.github.io/blog/automating-template-builds-with-packer.html

https://www.virtuallyghetto.com/2018/04/new-instant-clone-architecture-in-vsphere-6-7-part-1.html

https://damgoodadmin.com/2017/11/05/fully-automate-software-update-maintenance-in-cm/

https://www.adamtheautomator.com/use-powershell-automate-log-reviews/

https://www.adamtheautomator.com/creating-first-docker-windows-server-container/

https://blogs.vmware.com/PowerCLI/2018/03/powercli-docker-image-updated.html

https://blogs.vmware.com/code/2018/03/05/getting-started-vsphere-automation-sdk-net/

Docker on Windows 10, are containers for me?

https://robertsmit.wordpress.com/2018/02/21/clustering-fileserver-data-deduplication-on-windows-2016-step-by-step-sofs-winserv-refs-windowsserver2016-dedupe/

 

Posted in AWS, VMware | Tagged , | Leave a comment

Tips to edit the .vmdk (descriptor file)

Posted on November 30, 2018 by

We are in the situation to change the scsi adapter detail in the disk vmdk (descriptor file) and the count of the VM list is very high so used the below steps to change it on the multiples VMs.

Log-in to the SSH of the ESXi

Check what all vmdk having lsisas1068

 grep lsisas1068 /vmfs/volumes/storage/*/????????????????????????????????.vmdk

Based on the VM Name count ( ???????????????????????????????? )

Copy the vmdk file to /tmp/backup

 cp/vmfs/volumes/storage/*/????????????????????????????????.vmdk /tmp/backup/

Check and Apply the Changes

 sed ‘s/lsisas1068/lsilogic/’ /vmfs/volumes/storage/*/????????????????????????????????.vmdk

sed -i ‘s/lsisas1068/lsilogic/’ /vmfs/volumes/storage/i-2358-666282-VM/????????????????????????????????.vmdk

Verify the changes

diff /tmp/filename /vmfs/volumes/storage/i-2358-666282-VM/????????????????????????????????.vmdk

grep lsisas1068 /vmfs/volumes/storage/*/????????????????????????????????.vmdk

Posted in ESX command, ESXi issue, VMware | Tagged , | Leave a comment

Tips to update the certificate request with SANs (subject alternative names)

Posted on October 31, 2018 by

When we deploy our custom certificate in the https://<vrops>/admin UI, vROps will take care of distribute that same certificate to all vROps nodes.

Because vROps uses one certificate for all nodes, we must have every nodes’ IP address and FQDN (and short name, if your systems support short names) as Subject Alternate Names in the one certificate.On top of that we need to add the alias name to the master node to the SAN so that when we accessing the like with URL it will come with the certificate.

I have found the below two links which helped to achieve the same.

https://geekflare.com/san-ssl-certificate/ https://gist.github.com/croxton/ebfb5f3ac143cd86542788f972434c96

https://docs.vmware.com/en/vRealize-Operations-Manager/7.0/com.vmware.vcom.config.doc/GUID-F7DF7AFA-E32D-49AF-8E3F-06A807E65D89.html.

 

 

 

Posted in Certificate, VMware, vROPs | Tagged , , , | Leave a comment

Inactive VMs in SRM recovery.

Posted on September 30, 2018 by

We were testing the SRM recovery and noticed few VMs are skipped during the testing without any error and when we generated the logs ,skipped  VMs were showing as inactive.

Screen Shot 2018-09-30 at 3.37.44 PM

After few investigation we have noticed that the Guest OS was selected in the VM level as other , while the VMs which is recovered without any issue was in proper OS family.Issue got resolved and able to recover once we edited the  VM to the proper OS.

Screen Shot 2018-09-30 at 3.59.29 PM

Interesting part is same VMs were success during our last year DR-Test and it was running VC6.0u3\SRM6.1.1 and upgraded now to VC6.5u2\SRM6.5.I think something might have changed in the 6.5 version which required Guest-OS details to be placed properly ..

Posted in SRM, VMware | Tagged , , , , | Leave a comment