Improving FreeNAS iSCSI Performance with VMware ESXi

What does the IOPS setting do?

Included in every version of vSphere is Native Multi Pathing (NMP).  NMP offers the Round Robin Path Selection Policy (PSP) which works by utilising all available active paths to the datastores.  Round Robin sends all IOPS down a single active path until the number of IOPS sent down that path reaches the limit defined by the “IOOperation Limit” setting before it moves on to the next available active path.  Once all active paths have been used then it moves back to the first path and the cycle starts again.

Why change the Round Robin IOPS setting?

By default the Round Robin PSP will send 1000 IOPS down each path before moving on to the next one.  Tweaking this setting correctly can achieve a notable increase in disk IO throughput by more effectively utilizing the available paths.

For those of you that would like to know more, I have provided a link to an interesting article that explores the difference in performance between certain “IOOperation Limit” settings on various P2000 G3 devices here.

How do you change the Round Robin IOOperation Limit setting?

The Round Robin “IOOperation Limit” setting can be managed using VMware vSphere CLI.The commands are slightly different on an ESXi 5.x host compared to an ESXi 4.x host.

Note: A problem was identified whereby the Round Robin IOPS setting would be reset to default on ESXi hosts following a reboot.  This was resolved in ESXi 4.0 U2.

The commands for ESXi 4.x are:

Check the “IOOperation Limit” setting for a datastore:

esxcli nmp roundrobin getconfig -d [device UID]
Example output:
Set the “IOOperation Limit” setting for a datastore:
esxcli nmp roundrobin setconfig -d [device UID] -t iops -I [IOPS]
Example output:

The commands for ESXi 5.x are:

Check the “IOOperation Limit” setting for a datastore:

esxcli storage nmp psp roundrobin deviceconfig get -d [device UID]
Example output:
Set the “IOOperation Limit” setting for a datastore:
esxcli storage nmp psp roundrobin deviceconfig set -d [device UID] -t iops -I [IOPS]
Example output:

Ideally, you will need to run some tests on your own ESXi environment to establish the best [IOPS] value to use in your specific scenario.  This topic is, however, discussed in another of our blogs as mentioned in the section above.

The [device UID] can be obtained via the vSphere Client by following these steps:

1. Locate the Datastore that you want to set the “IOOperations Limit” setting for.  That can be via the “Summary” tab of a host / cluster or via the “Datastores” Invetory view etc.

2. Right-Click on the datastore and click on “Properties…” in the pop-up menu that apears

3. Look at the “Extents” section of the datastore properties box that appears and note the naa.600xxxxxx number that is displayed within the brackets.  This is the [device UID] that needs to be input into the esxcli command


Changing the setting from the default 1,000 IOPS to 1 IOPS yielded the following result in ATTO benchmarks. ATTO isn’t good at benchmarking ZFS storage arrays because of the ARC cache, but you can still see the substantial performance difference here. Whereas before, performance was hitting a wall curiously close to the 1Gb/s saturation limit it now manages to saturate 3 GigE links from the 2048MB test onwards.

Performance before set to default 1,000 IOPS
Performance after set to 1 IOPS.