Datastore with NFS4 slowness issue and Netapp\VMware findings.

Already mentioned in my previous blog here about the NFS 4 slowness issue compared to the NFS3 and we captured the packets and worked with the Netapp, below are the findings of our test when we tried ls -lahR | wc -l

  • There was no latency found in the perf-archives.
  • Took packet traces of nfsv3 and nfsv4 while listing the directory, 30 minutes apart.
  • There is a huge jump in lookup calls in V4.1.  From the V4.1 packet traces, even though readdir call is returning entries with file name FH and attributes, still the client is sending explicit lookup calls (i.e. compound with PUTFH,LOOKUP,GETFH,GETATTR) for directory entries.

    NFS3 SRT’s

    Index  Procedure/Opcodes
    /Commands     Calls  Min SRT Max SRT Avg SRT Sum SRT
    1      GETATTR 55     0.000059      0.004604      0.000414       0.022770
    3      LOOKUP 14363  0.000060      0.471455      0.000318       4.565892 <<<
    4      ACCESS 2874   0.000048      0.012290      0.000353       1.015239
    17     READDIRPLUS   6275   0.000072      0.922698       0.001608      10.091936 <<<
    18     FSSTAT 3      0.000065      0.004150      0.001443       0.004329


    NNFS 4.1 SRT’s
    Index  Procedure/Opcodes
    /Commands     Calls  Min SRT Max SRT Avg SRT Sum SRT
    1      COMPOUND (proc #)    95810  0.000020      1.020477       0.001411      135.205752
    3      ACCESS 2887   0.000068      0.004859      0.000321       0.925366
    9      GETATTR 95788  0.000068      1.020477      0.001411       135.198447
    10     GETFH  86659  0.000112      1.020477      0.001164       100.898088
    15     LOOKUP 80923  0.000094      1.020477      0.001144       92.574459 <<<
    16     LOOKUPP 5754   0.000123      0.845588      0.001448       8.330835

    22     PUTFH  95806  0.000068      1.020477      0.001411       135.205653
    26     READDIR 6233   0.000147      0.900400      0.005354       33.373798   <<<
    53     SEQUENCE      95810  0.000020      1.020477       0.001411      135.205752

We decided to test it on the same single datastore by mounting it as NFS 3 and capturing the data, once the test is done then unmount it and mount it as NFS 4 so on the same datastore with the same size and data below are the result.

Based on the result, VMware will be working on the option to enable the Lookup cache and mostly it will be available in future ESXi patches.

This entry was posted in ESX command, ESXi issue, ESXi Patches, VMware and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s