VMWare ATS locking causes LUNs to be unwritable

Adding, accessing LUNs, creating VMs or scanning and refreshing HBAs (hw iscsi) on ESX result in errors on the storage system:

idm: [ID 465152 kern.warning] WARNING: idm_refcnt_rele: refobj ffffff228be9f000, depth: 8, stacks(14): 0xfffffffff832fa0b, 0xfffffffff8377bc4, 0xfffffffff83792b6, 0xfffffffff8377470, 0xfffffffff8330ab2, 0xfffffffff83305ea, 0xfffffffff8338f1b, 0xfffffffffb86ea08,

vmkernel.log during vmhba rescan

cpu4:33303)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4130804461c0, 35530) to dev "naa.6000402e50000000482a4098f8184efd" on path "vmhba2:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2015-01-21T14:07:43.405Z cpu4:33303)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000402e50000000482a4098f8184efd" state in doubt; requested fast path state update...

vmkernel log during vmhba update

cpu5:32833)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000402e50000000482a4098f8184efd" state in doubt; requested fast path state update...
 cpu5:32833)ScsiDeviceIO: 2337: Cmd(0x413080441e40) 0x89, CmdSN 0x4d from world 1303441 to dev "naa.6000402e50000000482a4098f8184efd" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
cpu4:1303441)FSAts: 1184: vol 'datastore1', lock at 4331520: Unlocking lock failed: Timeout:

~ # esxcfg-rescan -u vmhba2
Error: Not all VMFS volumes were updated; the error encountered was 'Timeout'.


Setting pulse [HB state abcdef02 offset 3939840 gen 1 stampUS 488532767167 uuid 54b84c8e-953eb25c-44aa-001e675af992 jrnl <FB 0> drv 14.60] on vol 'datastore1' failed: Timeout

This is due to a problem where ATS only is not properly supported by the storage system.
ATS-only is enabled on the LUN (vmfs5 only)

# vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

You see output similar to:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only

# esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLockingYou see output similar to this:

Path: /VMFS3/HardwareAcceleratedLocking
   Type: integer
   Int Value: 1

The Int Value of 1 is indicating that VAAI features are enabled.

Disable ATS locking:

esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking


The Int Value of 0 is indicates that VAAI features are disabled.

rescan all the adapters and the errors should no longer appear.
The LUNs can now also be used for adding VMs and creating files / directories.

There are 2 ways to disable ATS



-      Per datastore

# vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxx

-      Per esxi host

# esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking

To re-enable just set 1

To check if it’s enabled or disabled:

vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

You see output similar to:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only

Enabling/disabling takes effect immediately without requiring a reboot.
If VMs are running on the datastore, the setting can’t be changed, until all VMs are stopped or moved.