Recently we have noticed in a few datastore the search operation is taking more time than expected and in our testing, it was identified that compare to NFSv4, NFSv3 results are better.

Our testing is to first list files from the host shell and we mounted the same storage as nfsv4 and nfsv3 to the same host. We ran this command from the host shell against both storages. time ls -lahR | wc -l . nfsv4 takes 1 minute and 30 seconds to finish running this command. When it is nfsv3, it only takes 15 seconds. 

Then tried using Powershell. Use the below way to search files. 

$searchspec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec

$searchSpec.matchpattern = “*.vmx”

$taskMoRef = $dsBrowser.SearchDatastoreSubFolders_Task($datastorePath2, $searchSpec) 

The data transfer rate is pretty much the same but the slowness issue is at the list/search file for nfs4.

Based on the results we involved the vendors to investigate the issue. NetApp conducted a thorough investigation and determined there were no performance issues, but VMware acknowledged there was a problem with NFS v4.1. Below was the reply from the VMware.

“Based on the analysis, Engineering team has identified that the issue related to slow search on NFSv4.1 is caused because the NFSv4.1 does not support “Directory Name Lookup Cache(DNLC) yet. However, for NFSv3, most of the LOOKUP calls are served from cache, which avoids sending a LOOKUP instruction to the NFS server. The VMware engineering is working to add this feature for NFSv4.1 however we do not have a version confirmation where this is expected to be included.”

We applied the latest patches but the issue still exists and hopefully, in the future patch it will get fixed.

Update: OCT2022

For more details check the blog.