vSphere client and Powercli fails to connect vCenter after TLSv1.0 disabled.

Posted on March 20, 2017 by

As per the KB 2148819 , TLSv1.0 has been disabled from the VC and issue started to connect the VC from the desktop client and also using the powercli .

Error from the Desktop client:

Error from the Powercli

Using the web-client there is no issue on connecting the vcenter or ESX hosts and only the issue is from the desktop client and powercli . After reading the KB again noticed that they already mentioned in notes about the issue and pointed to the another KB 2149000 which describes the issue and add to do few changes on the below file with few MS .Net patches

 C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure  Client\Launcher\VpxClient.exe.config

Edit the VpxClient.exe.config file by setting the parameters

<add key = "EnableTLS12" value =  "false" /> as
<add key = "EnableTLS12" value =  "true" />

After doing the changes also had the same issue and finally it got resolved by re-installing the desktop client.

But still connecting the vCenter using the powercli was not fixed and finally found the another KB 2137109 which asked to do the below registry changes which fixed the issue.

Must use PowerCLI 6.0 R1 or later. Earlier versions of PowerCLI work with versions of the .NET Framework that cannot use the TLSv1.1 and TLSv1.2 protocols by editing the registry.
  • For 32-bit processes, change the following registry key value to 1.

    Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\[.NET_version]
    Value: SchUseStrongCrypto (DWORD)

  • For 64-bit processes, in addition to the above registry key, change the following registry key value to 1.

    Key: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\[.NET_version]
    Value: SchUseStrongCrypto (DWORD)

Reference :

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2137109

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2149000

Advertisements
Posted in ESXi issue, vCSA 6.0, VMware | Tagged , , | Leave a comment

Primary\Secondary DNS IP Fail-Over bug in VMware vCenter Server Appliance 6.0 Update 2 ( VCSA U2)

Posted on February 20, 2017 by

We have the PRD setup with external PSC and VC which is configured with the Primary DNS and Secondary DNS . Due to the hardware issue on our primary DNS server , it went down and we couldn’t connect the VC.

All other application in our environment was working fine and we login to the PSC and VC with the port 5480 ( https://VC:5480 ) and manually changed the primary DNS IP to the working DNS server and within few seconds , VC started connecting to the PSC and allowing the AD authentication .

In our investigation we couldn’t find any concert reason for the failure and also tested in lab by just changing the Primary DNS to some unknown IP and didnt find any issue on the connectivity .

Finally raised the ticket with VMware and they confirmed that the issue is because of some bug in the VCSA Update 2 and they are working on to fix the issue in the next update 3 and also they confirmed it has been fixed in the VCSA 6.5 version but still no answer for my lab  environment which is working fine on changing the primary DNS.

UPDATE 3/16/2017 : VC 6.0 U3 release notes doesn’t show anything related to this bug fix and when we checked with VMware they confirmed still it is in testing stage and not included in the latest U3 update..

Also pls find the blog which list all the known issue on the VCSA

 

Posted in vCSA 6.0, VCSA6.5 | Tagged , , , | 2 Comments

Useful information and links about Microsoft Remote Procedure Call (RPC)

Posted on January 28, 2017 by

The diagram below shows the RPC workflow starting with the registration of the server application with the RPC Endpoint Mapper (EPM) in step 1 to the passing of data from the RPC client to the client application in step 7.

rpc

  1. Server app registers its endpoints with the RPC Endpoint Mapper (EPM)
  2. Client makes an RPC call (on behalf of a user, OS or application initiated operation)
  3. Client side RPC contacts the target computers EPM and ask for the endpoint to complete the client call
  4. Server Machine’s EPM responds with an endpoint
  5. Client side RPC contacts the server app
  6. Server app executes the call, returns the result to the client RPC
  7. Client side RPC passes the result back to the client app

How RPC Works

https://technet.microsoft.com/en-us/library/cc738291(v=ws.10).aspx

Troubleshooting “RPC server is unavailable” error, reported in failing AD replication scenario.

https://blogs.technet.microsoft.com/abizerh/2009/06/11/troubleshooting-rpc-server-is-unavailable-error-reported-in-failing-ad-replication-scenario/

Restricting Active Directory RPC traffic to a specific port

https://support.microsoft.com/en-us/kb/224196

How to configure RPC dynamic port allocation to work with firewalls

https://support.microsoft.com/en-in/kb/154596

https://blogs.technet.microsoft.com/askpfeplat/2015/01/11/rpc-endpoint-mapper-returns-dynamic-port-incorrectly-when-active-directory-is-configured-to-use-static-port/

Have you set static port on the DC for netlogon or for any other interfaces?

Long logon time after you set a specific static port for NTDS and NETLOGON in a Windows Server 2008 R2-based domain environment

http://support.microsoft.com/kb/2827870/en-us

AD replication fails with an RPC issue after you set a static port for NTDS in a Windows-based domain environment

http://support.microsoft.com/kb/2912805/en-us

Logon fails after you restrict client RPC to DC traffic in Windows Server 2012 R2 or Windows Server 2008 R2

http://support.microsoft.com/kb/2987849/en-us

Use the script https://gallery.technet.microsoft.com/Test-RPC-Testing-RPC-4396fcda that helps to test the RPC connectivity via TCP: This script tests TCP network connectivity to not just the RPC Endpoint Mapper on port 135, but it also checks TCP network connectivity to each of the registered endpoints returned by querying the EPM.  Many firewall teams have a difficult time with RPC, and they will end up allowing the Endpoint Mapper on port 135, but forget to also allow the ephemeral ports through the firewall.  This script uses localhost by default, but obviously you can specify a remote machine name or IP address to test a server across the network.  The script works by P/Invoking functions exported from rpcrt4.dll to get an enumeration of registered endpoints from the endpoint mapper, so it’s not just a wrapper around portqry.exe.

One of the issue if the ephemeral ports are blocked between clients and the domain controller it will show the RPC error while trying to join a client machine to the domain. Client gets joined to the domain and later fails with error “Changing the Primary Domain DNS name of this computer to “” failed. The name will remain “testlab.com. The error was: The RPC server is unavailable”.

use the below link to make sure we opened the required ports for the communication between clients and the DC.

How to configure a firewall for domains and trusts

https://support.microsoft.com/en-us/kb/179442

 

 

 

 

Posted in Windows | Tagged , , | Leave a comment

Useful links about Windows Failover Clustering.

Posted on December 15, 2016 by

Free Ebook

http://blogs.technet.com/b/askcore/archive/tags/failover+cluster/default.aspx

 

Understanding the Cluster Debug Log in 2008

http://blogs.technet.com/b/askcore/archive/2010/04/13/understanding-the-cluster-debug-log-in-2008.aspx

 

Troubleshooting Cluster Logs 101 – Why did the resources failover to the other node?

http://blogs.technet.com/b/askcore/archive/2008/02/06/troubleshooting-cluster-logs-101-why-did-the-resources-failover-to-the-other-node.aspx

 

Measuring Disk Latency with Windows Performance Monitor (Perfmon)

http://blogs.technet.com/b/askcore/archive/2012/02/07/measuring-disk-latency-with-windows-performance-monitor-perfmon.aspx

 

Planning Failover Cluster Node Sizing

http://blogs.msdn.com/b/clustering/archive/2014/07/03/10539483.aspx

 

Configuring Windows Failover Cluster Networks

http://blogs.technet.com/b/askcore/archive/2014/02/20/configuring-windows-failover-cluster-networks.aspx

 

Windows Server 2008 R2 Failover Clustering – Best Practice Guide

http://blogs.technet.com/b/aevalshah/archive/2012/05/15/windows-server-2008-r2-failover-clustering-best-practice-guide.aspx

 

Windows Server 2008 R2 Cluster: List of Hotfixes

http://social.technet.microsoft.com/wiki/contents/articles/2008.windows-server-2008-r2-cluster-list-of-hotfixes.aspx

 

What is RHS and what does it do?

http://blogs.msdn.com/b/clustering/archive/2009/06/27/9806160.aspx

 

Resource Hosting Subsystem (RHS) In Windows Server 2008 Failover Clusters

http://blogs.technet.com/b/askcore/archive/2009/11/23/resource-hosting-subsystem-rhs-in-windows-server-2008-failover-clusters.aspx

 

Understanding how Failover Clustering Recovers from Unresponsive Resources

http://blogs.msdn.com/b/clustering/archive/2013/01/24/10388009.aspx

 

978527 The Resource Hosting Subsystem (Rhs.exe) process stops unexpectedly when you start a cluster resource in Windows Server 2008 R2

https://support.microsoft.com/en-us/kb/978527

 

815267 How to enable User Mode Hang Detection on a server cluster in Windows Server 2003 and in Windows 2000 Server SP4

https://support.microsoft.com/kb/815267

 

Decoding Bugcheck 0x0000009E

http://blogs.msdn.com/b/clustering/archive/2013/11/13/10467483.aspx

 

Comparing Hotfixes across Multiple Node Failover Clusters

http://blogs.technet.com/b/askpfeplat/archive/2012/10/15/comparing-hotfixes-across-multiple-node-failover-clusters.aspx

 

Keep your Failover Clustering deployment healthy!

http://blogs.technet.com/b/mspfe/archive/2013/01/11/keep-your-failover-clustering-deployment-healthy.aspx

 

Video – https://mva.microsoft.com/en-us/training-courses/failover-clustering-in-windows-server-2012-r2-8489

Posted in SQL server Failover Cluster, Windows | Tagged , , | Leave a comment

Steps to upgrade the vCSA (PSC\VC ) 6.0 to 6.5.

Posted on November 16, 2016 by

Read the KB 2147548 \ KB 2147686 before upgrading the vshpere environment for the Compatibility considerations and also for the best practices to make sure all the products which is connected to the vsphere will support the 6.5 version.

Update 3/16/2017: As per the VC update 3 release notes “ Upgrading from vCenter Server 6.0 Update 3 to vCenter Server 6.5 is not supported”

Update 8/1/2017: We can now migrate VC6.0 update 3 to VC 6.5 U1

Once ISO is downloaded , open the content inside the folder and we can see the below folder structure and we have several ways to install or upgrade the VCSA 6.5.Basically we have below three types of the installation and Readme will give the clear instruction on the installation methods.

a) The UI based installer
b) The command-line installer
c) Migration from vCenter Server Windows to Appliance

First good factor on 6.5 is no need to have the  Client Integration Plugin and we can see UI Based Installer steps.

Before starting the upgrade, rename the existing  PSC\VC on the vcenter as old because during the upgrade process it will ask  for the target PSC\VC name for the deployment and if it is already on the same VC or ESX then it will show the error as VM name already exists in the vcenter.

Another important thing is we need one temporary IP for migrating the data from the OLD version to the new psc\VC 6.5.

One more item is we need to make sure to remove the DRS from fully automated during the upgrade process.

Upgrading the appliance is the two stage process . First stage involves deploying the new VM to the ESX\VC and the second process is copying the data from the source to destination appliance .

Open Windows Explorer and in the vCenter Server Appliance installation
directory, navigate to the ‘vcsa-ui-installer\win32’ folder.Click on the isntaller.exe

vc6-5

It will open the console to select for the new installation or upgrade.

vc6-51

Next it will show the option and process for upgrade

vc6-53

Accept the license agreement

vc6-54

Next provide the source PSC\VC IP and also select the ESX\VC which the source appliance is running.

wwe

ddd

Next provide the target ESXi\VC in which appliance to be deployed.

untitdled

Next have to provide the Target VM settings to be deployed so like mentioned if we are not renaming the existing VM name in the inventory then it will show the error as already VM name is exists ..

untitdlesd

So make sure the current appliance name is renamed in the inventory.

untieed

Select the datastore

und

Provide the temporary IP to copy the data and make sure it is in same vlan

vland

It will show all the configuration details .

vland_censored

It will go all the below process

untfed

It was in 99% for almost 10 mins while configuring the network and the first phase got completed.

On the second phase it will show the below warning to make sure the appliance is not under fully automated DRS to avoid vmotion of the VM to the other host.

titled

if all the configuration is correct then the PSC installation will be completed.

title1d

We can verify the PSC by login in to the https//PSC:5480 and alsp https://PSC/psc.

Also we can notice the old PSC will be in power-down in Vcenter.

Vcenter Appliance upgrade

Next we can see the vcenter upgrade and we need to follow the same steps as PSC so I will be covering only the second phase of the installation.

Once the first-phase is completed the second phase will start with the pre-upgrade steps .

vcd

Same like PSC it will show the warnings to remove fully automate DRS and also it will highlight to make sure other extension will work with the new upgrade .

war

Next it will show the data size to be copied to the new target server and the size will be depends on our environment.Also we can decide what data we need to copy to the new upgrade target VM.

ddl

It will show the source and target vcenter and its configuration.

unt0d

It will show the warning that source VCenter will be shoutdown.

stitled

upgrade process will be started.

lntitled

We can see the status of copying the data and setting the target VM.

lntif

lndtif

ded

So once the installtion is completed we can take the vcenter using the web and pls note from the Vsphere 6.5 we cant use the desktop C# client to access the vcenter.

 

 

 

 

 

 

 

 

 

 

 

 

 

Posted in Vcenter Appliance, vCSA 6.0, VCSA6.5, VMware | Tagged , , | Leave a comment

vdcrepadmin to find the replication status and design the Platform Services Controller 6.0

Posted on October 31, 2016 by

After attending the VMworld PSC session , I was thinking about to test the VDCREPADMIN tool which helps to find the replication status and to re-desgin the PSC.

Currently we have three PSCs which connects to each other in an in-line fashion, with each PSC installed against the previous PSC, rather than a hub-and-spoke fashion where all of the PSCs would terminate to a central PSC or mesh topology.

 

psc

VDCREPADMIN Showservers is to displays all of the PSCs in a vSphere domain.

Login to the appliance and go to the below path

cd /usr/lib/vmware-vmdir/bin

Run this command to show all PSCs in the vSphere domain:

vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password

psc1

From the output we can see the PSC Names , Site and Domain.

VDCREPADMIN showpartners is to display the partner PSC.

vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

 

psc2_censored

from the out put we can find the partnership between the PSCs which was installed in an in-line fashion, with each PSC installed against the previous PSC

  • PSC35.* has a replication partnership with PSC236
  • PSC36.* has a replication partnership with both PSC35.* and PSC37.*
  • PSC37.* has a replication partnership with both PSC236

VDCREPADMIN showpartnerstatus is to display the current replication partner of the PSC and also the current replication status between the two nodes.

psc3

Pls note you have to run the showpartnerstatus from each PSC to list the exact partner list and status.

Also from the output we can find the current sync with all the replication partner with the curren update sequence number ( USN ) value and in case of any failure check the log /var/log/vmware/vmdird/vmdird-syslog.log

VDCREPADMIN Createagreement is to create the replication agreements between the PSC with the same vSphere domains and not  between disparate (separate) vSphere domains.

So in our example we are creating the agreement between PSC37 and PSC35 so that in case of PSC36 failure still we have the replication with other partner in the domain.

psc6

Before running the agreement check the current partner.

vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

psc4

Use the following command to create a new replication agreement between PSCs .

vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u administrator -w Administrator_Password

psc5

If we have more number of PSC then plan to have the mesh topology and by using the createagreement we can plan the same. Due to replication time, it may take a few seconds to minutes for a complete mesh topology to be configured.

VDCREPADMIN Removeagreement is to remove the agreement from the replication partner.

First check the current partnership from the specified PSC:

vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password 

Use the following command to remove an existing replication agreement between PSCs:

vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN -H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

psc7

Reference :

KB 2127057

INF8225  – VMworld

 

Posted in Install and Configure VMware vCSA 6.0, Platform Services Controller (PSC ), VC6.0 Appliance Installation Issue, Vcenter Appliance, vCSA 6.0, VMware | Tagged , , , , , | 1 Comment

MLAG issue on CISCO UCS Servers with Arista switch

Posted on September 10, 2016 by

We were trying to configure the MLAG on the Arista switch on each uplink group of the CISCO UCS ( UCSC-C240-M3s) server . Initially ESXi 6.0 was installed and when we trying to enable the LACP on the dVswitch , host got disconnected and also stooped pinging and once we disabled the LACP option it came online. VMware support was involved and asked to install all the firmwares but still we had the same issue.

As a trouble-shooting step we installed the windows OS and once we enable the LACP option in the windows NIC Teaming it started same issue as server got disconnected and also stopped pinging.

Another set of ESX servers with same MLAG configuration on the HP Blade Enclosure on Arista was working fine without any issue .

We involved Cisco for the server and Arista from the switch end for the support , from the Cisco we didnt get any proper respond , finally the issue got escalated to the Tier-1 support in Arista and they found the issue is when the Cisco server is configured for trunk mode, the server is sending LACP frames tagged with a vlan id of 0. Currently, in the version of code we are running on this arista switch platform, it’s dropping these frames as it’s not accepting tagged LACPBPDUs. In addition, when the server is configured as an access port, it is sending the LACP frames tagged with the access vlan specified. Even if there is no default vlan specified, the server is still sending the LACP frames tagged with vlan 0. It seems there is no way to have the Cisco server to send the LACP frames un-tagged like HP blade chassis, tintris which are already on mlag.

Requested the Cisco engineer to see the reason for the server is tagging the LACP frames but didnt get the answer so as per the Arista recommendation we upgraded the  latest code on the switch to Version4.16.7M which fixed the issue.

Reference :

https://eos.arista.com/vmware-esx-5-arista-lacp-guide/

Posted in Arista MLAG, CISCO UCS Servers, ESXi issue, Windows | Tagged , | 1 Comment